Fluorescence imaging in a light deficient environment

ABSTRACT

An endoscopic imaging system for use in a light deficient environment includes an imaging device having a tube, one or more image sensors, and a lens assembly including at least one optical elements that corresponds to the one or more image sensors. The endoscopic system includes a display for a user to visualize a scene and an image signal processing controller. The endoscopic system includes a light engine having an illumination source generating one or more pulses of electromagnetic radiation and a lumen transmitting one or more pulses of electromagnetic radiation to a distal tip of an endoscope.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application claims the benefit of U.S. Provisional PatentApplication No. 62/610,888, filed Dec. 27, 2017, titled “HYPERSPECTRALIMAGING IN A LIGHT DEFICIENT ENVIRONMENT,” and claims the benefit ofU.S. Provisional Patent Application No. 62/723,989, filed Aug. 28, 2018,titled “HYPERSPECTRAL IMAGING IN A LIGHT DEFICIENT ENVIRONMENT,” whichare incorporated herein by reference in their entireties, including butnot limited to those portions that specifically appear hereinafter, theincorporation by reference being made with the following exception: Inthe event that any portion of the above-referenced provisionalapplications are inconsistent with this application, this applicationsupersedes the above-referenced provisional applications.

STATEMENT REGARDING FEDERALLY SPONSORED RESEARCH AND DEVELOPMENT

Not applicable.

BACKGROUND

Advances in technology have provided advances in imaging capabilitiesfor medical use. An endoscope may be used to look inside a body andexamine the interior of an organ or cavity of the body. Endoscopes maybe used for investigating a patient's symptoms, confirming a diagnosis,or providing medical treatment. A medical endoscope may be used forviewing a variety of body systems and parts, including for example, thegastrointestinal tract, the respiratory tract, the urinary tract, theabdominal cavity by way of a small incision, and so forth. Endoscopesmay further be used for surgical procedures, such as plastic surgeryprocedures, procedures performed on joints or bones, proceduresperformed on the neurological system, procedures performed within theabdominal cavity, and so forth.

Endoscopes have also been used in non-medical fields for viewing andinspecting spaces that may be inaccessible or difficult to see. Forexample, endoscopes may be used by planners or architects forvisualizing scale models of proposed buildings or cities. Endoscopes maybe used for visualizing an internal space of a complex system such as acomputer. Endoscopes may even be used by law enforcement or militarypersonnel for conducting surveillance in tight spaces or examiningexplosive devices.

Among the various uses for endoscopes, it may be beneficial to view aspace in color. A digital color image may include at least three layers,or “color channels,” for each pixel of the image. Each of the colorchannels measures the intensity and chrominance of light for a spectralband. Commonly, a digital color image includes a color channel for red,green, and blue spectral bands of light (this may be referred to as anRGB image). Each of the red, green, and blue color channels includebrightness information for the red, green, or blue spectral band oflight. The brightness information for the separate red, green, and bluelayers may be combined to create a digital color image. Because a colorimage is made up of separate layers, a digital camera image sensorcommonly includes a color filter array that permits red, green, and bluevisible light wavelengths to hit selected pixel sensors. Each individualpixel sensor element is made sensitive to red, green, or bluewavelengths and will only return image data for that wavelength. Theimage data from the total array of pixel sensors is combined to generatethe RGB image.

In the case of endoscopic imaging for medical diagnostics or medicalprocedures, it may be beneficial or even necessary to view a body cavitywith color images. For example, if an endoscope is used to view theabdominal cavity of a body, a color image may provide valuableinformation to help identify different organs or tissues within theabdomen, or to identify certain conditions or diseases within the space.As discussed above, a digital camera capable of capturing color imagesmay have at least three distinct types of pixel sensors to individuallycapture the red, green, and blue layers of the color images. The atleast three distinct types of pixel sensors may consume a relativelysignificant physical space (when compared with a color-agnostic pixelarray) such that the complete pixel array cannot fit on the small distalend of the endoscope that is inserted into the body. Because colordigital cameras may include the at least three distinct types of pixelsensors, the total pixel array (i.e. the image sensor) is commonlylocated in a hand-piece unit of an endoscope that is held by anendoscope operator and is not placed within the body cavity. For such anendoscope, light is transmitted along the length of the endoscope fromthe hand-piece unit to the distal end of the endoscope that is placedwithin the body cavity. This endoscope configuration has significantlimitations. Endoscopes with this configuration are delicate and can beeasily misaligned or damaged when bumped or impacted during regular use.This can significantly degrade the quality of the images generated bythe endoscope and necessitate that the endoscope be frequently repairedor replaced.

In some cases, and particularly in the case of medical imaging ormedical procedures, it may be beneficial to see more than a color image.Color images reflect what the human eye detects when looking at anenvironment. However, the human eye is limited to viewing only visiblelight and cannot detect other wavelengths of the electromagneticspectrum. At other wavelengths of the electromagnetic spectrum beyondthe “visible light” wavelengths, additional information may be obtainedabout an environment. One method of detecting additional informationabout an environment, beyond what the human eye is capable of detecting,is the use of fluorescent reagents. In the case of imaging for medicalpurposes, fluorescent reagents may provide a unique view of a bodycavity that highlights certain tissues, structures, or conditions thatthe human eye or a computer program cannot detect in an RGB image.

Fluorescence is the emission of light by a substance that has absorbedlight or other electromagnetic radiation. Certain fluorescent materialsmay “glow” or emit a distinct color that is visible to the human eyewhen the fluorescent material is subjected to ultraviolet light or otherwavelengths of electromagnetic radiation. Certain fluorescent materialswill cease to glow nearly immediately when the radiation source stops.

Fluorescence occurs when an orbital electron of a molecule, atom, ornanostructure is excited by light or other electromagnetic radiation,and then relaxes to its ground state by emitting a photon from theexcited state. The specific frequencies of electromagnetic radiationthat excite the orbital electron, or are emitted by the photon duringrelaxation, are dependent on the particular atom, molecule, ornanostructure. In most cases, the light emitted by the substance has alonger wavelength, and therefore lower energy, than the radiation thatwas absorbed by the substance. However, when the absorbedelectromagnetic radiation is intense, it is possible for one electron toabsorb two photons. This two-photon absorption can lead to emission ofradiation having a shorter wavelength, and therefore higher energy, thanthe absorbed radiation. Additionally, the emitted radiation may also bethe same wavelength as the absorbed radiation.

Fluorescence imaging has numerous practical applications, includingmineralogy, gemology, medicine, spectroscopy for chemical sensors,detecting biological processes or signals, and so forth. Fluorescencemay particularly be used in biochemistry and medicine as anon-destructive means for tracking or analyzing biological molecules.The biological molecules, including certain tissues or structures, maybe tracked by analyzing the fluorescent emission of the biologicalmolecules after being excited by a certain wavelength of electromagneticradiation. However, relatively few cellular components are naturallyfluorescent. In certain implementations, it may be desirable tovisualize a certain tissue, structure, chemical process, or biologicalprocess that is not intrinsically fluorescent. In such animplementation, the body may be administered a dye or reagent that mayinclude a molecule, protein, or quantum dot having fluorescentproperties. The reagent or dye may then fluoresce after being excited bya certain wavelength of electromagnetic radiation. Different reagents ordyes may include different molecules, proteins, and/or quantum dots thatwill fluoresce at particular wavelengths of electromagnetic radiation.Thus, it may be necessary to excite the reagent or dye with aspecialized band of electromagnetic radiation to achieve fluorescenceand identify the desired tissue, structure, or process in the body.

Fluorescence imaging may provide valuable information in the medicalfield that may be used for diagnostic purposes and/or may be visualizedin real-time during a medical procedure. Specialized reagents or dyesmay be administered to a body to fluoresce certain tissues, structures,chemical processes, or biological processes. The fluorescence of thereagent or dye may highlight body structures such as blood vessels,nerves, particular organs, and so forth. Additionally, the fluorescenceof the reagent or dye may highlight conditions or diseases such ascancerous cells or cells experiencing a certain biological or chemicalprocess that may be associated with a condition or disease. Thefluorescence imaging may be used in real-time by a medical practitioneror computer program for differentiating between, for example, cancerousand non-cancerous cells during a surgical tumor extraction. Thefluorescence imaging may further be used as a non-destructive means fortracking and visualizing over time a condition in the body that wouldotherwise not be visible by the human eye or distinguishable in an RGBimage.

However, fluorescence imaging requires specialized emissions ofelectromagnetic radiation and may further require specialized imagingsensors capable of reading the wavelength of electromagnetic radiationthat is emitted by the fluoresced structure or reagent. Differentreagents or dyes may be sensitive to different wavelengths ofelectromagnetic radiation and may further emit different wavelengths ofelectromagnetic radiation when fluoresced. Imaging systems may then behighly specialized and tuned for a certain reagent or dye such that thesystem is configured to emit certain wavelengths of electromagneticradiation and includes imaging sensors configured for reading certainwavelengths of electromagnetic radiation. Such imaging systems may beuseful for very limited applications and may not be capable offluorescing more than one reagent or structure during a single imagingsession. It can be very costly to need multiple distinct imaging systemsthat are each configured for fluorescing a particular reagent or dye.Additionally, it may be desirable to administer multiple reagents ordyes that are each configured for fluorescing a different structure orcondition and visualizing the fluorescence imaging for each of thereagents or dyes in a single overlaid image.

Further, it may be desirable to overlay fluorescence imaging on ablack-and-white or color image to provide context to a practitioner orcomputer algorithm. Historically, this would require the use of a camera(or multiple cameras) having many distinct types of pixel sensors thatare each sensitive to distinct ranges of electromagnetic radiation. Thismay include the three separate types of pixels sensors for generating anRGB color image by way of conventional methods, along with additionalpixel sensors for generating the fluorescence image data at differentwavelengths of the electromagnetic spectrum. This may consume arelatively large physical space and necessitate a large pixel array toensure that image resolution is satisfactory. In the case of endoscopicimaging, the camera or cameras may be placed in an endoscope hand-unitor robotic-unit because the multiple wavelength-sensitive pixel sensorsrequire too much physical space, and necessitate too large a pixelarray, to be placed at the distal end of the endoscope within the bodycavity. This introduces the same disadvantages mentioned above and cancause the endoscope to be very delicate such that image quality issignificantly degraded when the endoscope is bumped or impacted duringuse.

This disclosure relates generally to electromagnetic sensing and sensorsthat may be applicable to endoscope imaging. The disclosure also relatesto low energy electromagnetic input conditions as well as low energyelectromagnetic throughput conditions. The disclosure relates moreparticularly, but not necessarily entirely, to a system for producing animage in light deficient environments and associated structures, methodsand features, which may include controlling a light source throughduration, intensity or both, pulsing a component controlled light sourceduring the blanking period of an image sensor, maximizing the blankingperiod to allow optimum light, and maintaining color balance.

The features and advantages of the disclosure will be set forth in thedescription which follows, and in part will be apparent from thedescription, or may be learned by the practice of the disclosure withoutundue experimentation. The features and advantages of the disclosure maybe realized and obtained by means of the instruments and combinationsparticularly pointed out in the appended claims.

BRIEF DESCRIPTION OF THE DRAWINGS

Non-limiting and non-exhaustive implementations of the disclosure aredescribed with reference to the following figures, wherein likereference numerals refer to like parts throughout the various viewsunless otherwise specified. Advantages of the disclosure will becomebetter understood with regard to the following description andaccompanying drawings where:

FIG. 1 is a schematic view of an embodiment of a system of a pairedsensor and an electromagnetic emitter in operation for use in producingan image in a light deficient environment, according to one embodiment;

FIG. 2 is a schematic view of complementary system hardware;

FIGS. 2A to 2D are illustrations of the operational cycles of a sensorused to construct one image frame, according to embodiments of thedisclosure;

FIG. 3 is a graphical representation of the operation of an embodimentof an electromagnetic emitter, according to one embodiment;

FIG. 4 is a graphical representation of varying the duration andmagnitude of the emitted electromagnetic pulse to provide exposurecontrol, according to one embodiment;

FIG. 5 is a graphical representation of an embodiment of the disclosurecombining the operational cycles of a sensor, the electromagneticemitter and the emitted electromagnetic pulses of FIGS. 2A-4, whichdemonstrate the imaging system during operation, according to oneembodiment;

FIG. 6 illustrates a schematic of two distinct processes over a periodof time from t(0) to t(1) for recording a frame of video for fullspectrum light and partitioned spectrum light, according to oneembodiment;

FIGS. 7A-7E illustrate schematic views of the processes over an intervalof time for recording a frame of video for both full spectrum light andpartitioned spectrum light in accordance with the principles andteachings of the disclosure;

FIGS. 8-12 illustrate the adjustment of both the electromagnetic emitterand the sensor, wherein such adjustment may be made concurrently in someembodiments in accordance with the principles and teachings of thedisclosure;

FIGS. 13-21 illustrate sensor correction methods and hardware schematicsfor use with a partitioned light system, according to embodiments of thedisclosure;

FIGS. 22-23 illustrate method and hardware schematics for increasing thedynamic range within a closed or limited light environment, according toembodiments of the disclosure;

FIG. 24 illustrates the impact on signal to noise ratio of the colorcorrection for a typical Bayer-based sensor compared with no colorcorrection;

FIG. 25 illustrates the chromaticity of 3 monochromatic lasers comparedto the sRGB gamut;

FIGS. 26-27B illustrate method and hardware schematics for increasingthe dynamic range within a closed or limited light environment,according to embodiments of the disclosure;

FIGS. 28A-28C illustrate the use of a white light emission that ispulsed and/or synced with a corresponding color sensor, according toembodiments of the disclosure;

FIGS. 29A and 29B illustrate an implementation having a plurality ofpixel arrays for producing a three-dimensional image, according toembodiments of the disclosure;

FIGS. 30A and 30B illustrate a perspective view and a side view,respectively, of an implementation of an imaging sensor built on aplurality of substrates, wherein a plurality of pixel columns formingthe pixel array are located on the first substrate and a plurality ofcircuit columns are located on a second substrate and showing anelectrical connection and communication between one column of pixels toits associated or corresponding column of circuitry;

FIGS. 31A and 31B illustrate a perspective view and a side view,respectively, of an implementation of an imaging sensor having aplurality of pixel arrays for producing a three-dimensional image,wherein the plurality of pixel arrays and the image sensor are built ona plurality of substrates;

FIGS. 32-36 illustrate embodiments of emitters comprising variousmechanical filter and shutter configurations, according to embodimentsof the disclosure;

FIG. 37 is a schematic diagram illustrating a system for providingillumination to a light deficient environment, according to oneembodiment;

FIG. 38 is a schematic block diagram illustrating a light source havinga plurality of emitters, according to one embodiment;

FIG. 39 is a schematic block diagram illustrating a light source havinga plurality of emitters, according to another embodiment;

FIG. 40 is a schematic block diagram illustrating a light source havinga plurality of emitters, according to yet another embodiment;

FIG. 41 is a schematic diagram illustrating a single optical fiberoutputting via a diffuser at an output to illuminate a scene, accordingto one embodiment;

FIG. 42 is a block diagram illustrating generating a filtered imageusing a filter, according to one embodiment;

FIG. 43 illustrates a portion of the electromagnetic spectrum dividedinto a plurality of different sub-spectrums which may be emitted byemitters of a light source, according to one embodiment;

FIG. 44 is a schematic diagram illustrating a timing diagram foremission and readout for generating a multispectral or hyperspectralimage, according to one embodiment;

FIG. 45 is a block diagram illustrating generating a filtered imageusing a filter, according to one embodiment;

FIG. 46 is a block diagram illustrating generating a filtered imageusing a plurality of filters, according to one embodiment;

FIG. 47 is a schematic diagram illustrating a grid array for objectand/or surface tracking, according to one embodiment;

FIG. 48 is a schematic flow chart diagram illustrating a method foremission and readout for generating a multispectral or hyperspectralimage, according to one embodiment; and

FIG. 49 is a schematic flow chart diagram illustrating a method foremission and readout for generating a fluorescence image, according toone embodiment.

DETAILED DESCRIPTION

The disclosure extends to methods, systems, and computer-based productsfor digital imaging that may be primarily suited to medical applicationssuch as medical endoscopic imaging. Such methods, systems, andcomputer-based products as disclosed herein may provide imaging ordiagnostic capabilities for use in medical robotics applications, suchas the use of robotics for performing imaging procedures, surgicalprocedures, and the like. In the following description of thedisclosure, reference is made to the accompanying drawings, which form apart hereof, and in which is shown by way of illustration specificimplementations in which the disclosure may be practiced. It isunderstood that other implementations may be utilized, and structuralchanges may be made without departing from the scope of the disclosure.

Endoscopes have a great variety of uses and may provide significantbenefits in the medical field. Endoscopy is used in medicine to lookinside a body and in some cases may provide imaging that would otherwisebe impossible to see or would require invasive surgical procedures.Endoscopes may be used for medical diagnostics, investigation, orresearch, and may also be used to perform medical procedures in aminimally invasive manner Medical endoscopes may provide significantbenefits to patients and medical practitioners by negating the need forpainful and invasive corrective or exploratory surgeries.

As disclosed herein, an endoscopic system for use in a light deficientenvironment, such as a cavity of a body, may include an imaging deviceand a light engine. The light engine may include an illumination sourcefor generating pulses of electromagnetic radiation and may furtherinclude a lumen for transmitting pulses of electromagnetic radiation toa distal tip of an endoscope. The lumen may transmit the pulses ofelectromagnetic radiation at particular wavelengths or bands ofwavelengths of the electromagnetic spectrum. The lumen may transmit suchpulses in a timed sequence and imaging data may be captured by a sensorduring each of the pulses. The imaging data associated with thedifferent wavelengths of the pulses may be used to generate a red greenblue (RGB) image and/or fluorescence images. In an embodiment,fluorescence imaging may be overlaid on a black-and-white or RGB image.

As disclosed herein, the systems, methods, and devices for an endoscopicimage system may provide specialized image data of a light deficientenvironment. The specialized image data may be used to generatefluorescence imaging and/or identify certain materials, tissues,components, or processes within a light deficient environment. Incertain embodiments, fluorescence imaging may be provided to apractitioner or computer-implemented program to enable theidentification of certain structures or tissues within a body. Suchfluorescence imaging data may be overlaid on black-and-white or RGBimages to provide additional information and context.

Further, such systems, methods, and devices for an endoscopic imagesystem may be used in coordination with certain reagents or dyes. In amedical imaging implementation, certain reagents or dyes may beadministered to a patient, and those reagents or dyes may fluoresce orreact to certain wavelengths of electromagnetic radiation. Theendoscopic image system as disclosed herein may transmit electromagneticradiation at specified wavelengths to fluoresce the reagents or dyes.The fluorescence of the reagents or dyes may be captured by an imagesensor to generate imaging to aid in the identification of tissues orstructures and/or to aid in diagnosis or research. In an implementation,a patient may be administered a plurality of reagents or dyes that areeach configured to fluoresce at different wavelengths and/or provide anindication of different structures, tissues, chemical reactions,biological processes, and so forth. In such an implementation, theendoscopic system as disclosed herein may emit each of the applicablewavelengths to fluoresce each of the applicable reagents or dyes. Thismay negate the historical need to perform individual imaging proceduresfor each of a plurality of reagents or dyes.

Medical endoscopes may provide a continuous digital image stream of aninterior space of a body where a distal end of the endoscope isinserted. In various implementations, it may be beneficial or evennecessary that the digital image stream provides full color imaging suchthat a medical practitioner may better differentiate between tissues andstructures in the body. In further implementations, it may be beneficialto provide hyperspectral imaging data to differentiate betweenstructures, tissues, processes, and conditions with enhanced precision.Additionally, hyperspectral imaging may enable a medical practitioner orcomputer program to receive information about a condition in a humanbody that is not visible to the human eye or discernable in an RGB colorimage.

Systems, methods, and devices are disclosed herein for generating colorimage data and/or fluorescence image data by an endoscope. A system ofthe disclosure includes an imaging device having a tube, one or moreimage sensors, and a lens assembly. The lens assembly may include atleast one optical element that corresponds to at least one of the one ormore image sensors. The system further includes a display to visual ascene and an image signal processing controller. The system may furtherinclude a light engine. The light engine includes an illumination sourceconfigured to generate one or more pulses of electromagnetic radiationand a lumen that transmits one or more pulses of electromagneticradiation to a distal tip of an endoscope. In an embodiment, at least aportion of the one or more pulses of electromagnetic radiation includesan excitation wavelength of electromagnetic radiation between 770 nm and790 nm that cause one or more reagents to fluoresce at a wavelength thatis different from the excitation wavelength of the portion of the one ormore pulses of electromagnetic radiation.

In an embodiment of the disclosure, an endoscope system illuminates asource and pulses electromagnetic radiation at a certain wavelength forexciting an electron in a reagent or dye. In an embodiment, the reagentor dye is configured to fluoresce in response to the certain wavelengthof electromagnetic radiation that is emitted by the endoscope system. Animage sensor in the endoscope system may read a fluorescence relaxationemission of the reagent or dye that may be of lower energy than thepulsed electromagnetic radiation for exciting the reagent or dye. Thereagent or dye may be specialized for labeling a certain tissue,structure, biological process, and/or chemical process.

Imaging reagents, including fluorescent reagents, may enhance imagingcapabilities in pharmaceutical, medical, biotechnology, diagnostic, andmedical procedure industries. Many imaging techniques such as X-ray,computer tomography (CT), ultrasound, magnetic resonance imaging (MRI),and nuclear medicine, mainly analyze anatomy and morphology and areunable to detect changes at the molecular level. Fluorescent reagents,dyes, and probes, including quantum dot nanoparticles and fluorescentproteins, may assist medical imaging technologies by providingadditional information about certain tissues, structures, chemicalprocesses, and/or biological processes that are present within theimaging region. Imaging using fluorescent reagents may enable celltracking and/or the tracking of certain molecular biomarkers.Fluorescent reagents may be applied for imaging cancer, infection,inflammation, stem cell biology, and others. Numerous fluorescentreagents and dyes are being developed and applied for visualizing andtracking biological processes in a non-destructive manner Suchfluorescent reagents may be excited by a certain wavelength or band ofwavelengths of electromagnetic radiation. Similarly, those fluorescentreagents may emit relaxation energy at a certain wavelength or band ofwavelengths when fluorescing, and the emitted relaxation energy may beread by a sensor to determine the location and/or boundaries of thereagent or dye.

In an embodiment of the disclosure, an endoscope system pulseselectromagnetic radiation for exciting an electron in a fluorescentreagent or dye. The wavelength or band of wavelengths of theelectromagnetic radiation may be particularly selected for fluorescing acertain reagent or dye. In an embodiment, the endoscope system may pulsemultiple different wavelengths of electromagnetic radiation forfluorescing multiple different reagents or dyes during a single imagingsession. A sensor of the endoscope system may determine a locationand/or boundary of a reagent or dye based on the relaxation emissions ofthe reagent or dye. The endoscope system may further pulseelectromagnetic radiation in red, green, and blue bands of visiblelight. The endoscope system may determine data for an RGB image and afluorescence image according to a pulsing schedule for the pulses ofelectromagnetic radiation.

In an embodiment of the disclosure, an endoscope system illuminates asource and pulses electromagnetic radiation for spectral orhyperspectral imaging. Spectral imaging uses multiple bands across theelectromagnetic spectrum. This is different from conventional camerasthat only capture light across the three wavelengths based in thevisible spectrum that are discernable by the human eye, including thered, green, and blue wavelengths to generate an RGB image. Spectralimaging may use any wavelength bands in the electromagnetic spectrum,including infrared wavelengths, the visible spectrum, the ultravioletspectrum, x-ray wavelengths, or any suitable combination of variouswavelength bands. Spectral imaging may overlay imaging generated basedon non-visible bands (e.g., infrared) on top of imaging based on visiblebands (e.g. a standard RGB image) to provide additional information thatis easily discernable by a person or computer algorithm.

Hyperspectral imaging is a subcategory of spectral imaging.Hyperspectral imaging includes spectroscopy and digital photography. Inan embodiment of hyperspectral imaging, a complete spectrum or somespectral information is collected at every pixel in an image plane. Ahyperspectral camera may use special hardware to capture any suitablenumber of wavelength bands for each pixel which may be interpreted as acomplete spectrum. The goal of hyperspectral imaging may vary fordifferent applications. In one application, the goal of hyperspectralimaging is to obtain the entire electromagnetic spectrum of each pixelin an image scene. This may enable certain objects to be found thatmight otherwise not be identifiable under the visible light wavelengthbands. This may enable certain materials or tissues to be identifiedwith precision when those materials or tissues might not be identifiableunder the visible light wavelength bands. Further, this may enablecertain processes to be detected by capturing an image across allwavelengths of the electromagnetic spectrum.

Hyperspectral imaging may provide particular advantages overconventional imaging in medical applications. The information obtainedby hyperspectral imaging can enable medical practitioners and/orcomputer-implemented programs to precisely identify certain tissues orconditions that may lead to diagnoses that may not be possible or may beless accurate if using conventional imaging such as RGB imaging.Additionally, hyperspectral imaging may be used during medicalprocedures to provide image-guided surgery that may enable a medicalpractitioner to, for example, view tissues located behind certaintissues or fluids, identify atypical cancerous cells in contrast withtypical healthy cells, identify certain tissues or conditions, identifycritical structures and so forth. Hyperspectral imaging may providespecialized diagnostic information about tissue physiology, morphology,and composition that cannot be generated with conventional imaging.

Endoscopic hyperspectral imaging may present advantages overconventional imaging in various applications and implementations of thedisclosure. In medical implementations, endoscopic hyperspectral imagingmay permit a practitioner or computer-implemented program to discern,for example, nervous tissue, muscle tissue, various vessels, thedirection of blood flow, and so forth. Hyperspectral imaging may enableatypical cancerous tissue to be precisely differentiated from typicalhealthy tissue and may therefore enable a practitioner orcomputer-implemented program to discern the boundary of a canceroustumor during an operation or investigative imaging. Additionally,hyperspectral imaging in a light deficient environment as disclosedherein may be combined with the use of a reagent or dye to enablefurther differentiation between certain tissues or substances. In suchan embodiment, a reagent or dye may be fluoresced by a specificwavelength band in the electromagnetic spectrum and therefore provideinformation specific to the purpose of that reagent or dye. The systems,methods, and devices as disclosed herein may enable any number ofwavelength bands to be pulsed such that one or more reagents or dyes maybe fluoresced at different times. In certain implementations, this mayenable the identification or investigation of a number of medicalconditions during a single imaging procedure.

A medical endoscope may pulse electromagnetic radiation at wavelengthbands outside the visible light spectrum to enable the generation ofhyperspectral images. Endoscopic hyperspectral imaging is a contactlessand non-invasive means for medical imaging that does not require apatient to undergo harmful radiation exposure common in other imagingmethods.

Conventional endoscopes, used in, for example, robotics endoscopicprocedures such as arthroscopy and laparoscopy, are designed such thatthe image sensors are typically placed within a hand-piece unit that isheld by an endoscope operator and is not inserted into a cavity. In sucha configuration, an endoscope unit transmits incident light along thelength of an endoscope tube toward the sensor via a complex set ofprecisely coupled optical components, with minimal loss and distortion.The cost of the endoscope unit is dominated by the optics because theoptics components are expensive, and the manufacturing process of theoptics components is labor intensive. Further, this type of endoscope ismechanically delicate and relatively minor impacts can easily damage thecomponents or upset the relative alignments of those components. Evenminor misalignments to the endoscope components (such as the preciselycoupled optical components) can cause significant degradation to imagequality or render the endoscope unusable. When the components aremisaligned, the incident light traveling along the length of theendoscope may fall off such that there is little or no light at thedistal end of the endoscope and the endoscope becomes unusable. Becauseconventional endoscopes require such precise and complex opticalcomponents, and because those components can become easily misaligned,such convention endoscopes require frequent and expensive repair cyclesto maintain image quality.

One solution to this issue is to place the image sensor within theendoscope itself at the distal end. Such a solution may eliminate theneed for a complex and precise collection of coupled optical componentsthat can be easily misaligned and/or damaged. This solution potentiallyapproaches the optical simplicity, robustness, and economy that isuniversally realized within, for example, mobile phone cameras. However,it should be appreciated that a great deal of the benefits provided byan endoscope arise from the compact size of the distal end of theendoscope. If the distal end of the endoscope is enlarged to accommodatethe multiple distinct wavelength-sensitive pixel sensors conventionallyused for color imaging or hyperspectral imaging, the pixel array may betoo large, and the endoscope may no longer fit in tight spaces or may beobstructive or invasive when used in a medical implementation. Becausethe distal end of the endoscope must remain very small, it ischallenging to place one or more image sensors at the distal end. Anacceptable solution to this approach is by no means trivial andintroduces its own set of engineering challenges, not the least of whichis the fact that the sensors for color and/or hyperspectral imaging mustfit within an area that is highly confined. This is particularlychallenging where a pixel array in conventional cameras include separatepixel sensors for each of red, green, and blue visible light bands,along with additional pixel sensors for other wavelength bands used forhyperspectral imaging. The area of the distal tip of the endoscope maybe particularly confined side-to-side in the X and Y dimensions, whilethere is more space along the length of the endoscope tube in the Zdimension.

Because many of the benefits of an endoscope arise from the small sizeof the distal end of the endoscope, aggressive constraints must beplaced on the image sensor area when the image sensors are located atthe distal end. These aggressive constraints placed on the sensor areanaturally results in fewer and/or smaller pixels within a pixel array.Lowering the pixel count may directly affect the spatial resolution,while reducing the pixel area may reduce the available signal capacityand thereby the sensitivity of the pixel, as well as optimizing thenumber of pixels such that image quality is maximized, the minimum pixelresolution and native number of pixels using the maximum pixel qualityand pitch, such that resolution is not an issue as well as lowering thesignal to noise ratio (SNR) of each pixel. Lowering the signal capacityreduces the dynamic range, i.e., the ability of the imaging device orcamera to simultaneously capture all the useful information from sceneswith large ranges of luminosity. There are various methods to extend thedynamic range of imaging systems beyond that of the pixel itself. All ofthem may have some kind of penalty, however, (e.g., in resolution orframe rate) and they can introduce undesirable artifacts, which becomeproblematic in extreme cases. Reducing the sensitivity has theconsequence that greater light power is required to bring the darkerregions of the scene to acceptable signal levels. Lowering the F-number(enlarging the aperture) can compensate for a loss in sensitivity, butat the cost of spatial distortion and reduced depth of focus.

In the sensor industry, complementary metal-oxide-semiconductor (“CMOS”)image sensors have largely displaced conventional charge-coupled device(“CCD”) image sensors in modern camera applications. CMOS image sensorshave greater ease of integration and operation, superior or comparableimage quality, greater versatility and lower cost, compared with CCDimage sensors. Typically, CMOS image sensors may include the circuitrynecessary to convert image information into digital data and havevarious levels of digital processing incorporated thereafter. This canrange from basic algorithms for the purpose of correctingnon-idealities, which may, for example, arise from variations inamplifier behavior, to full image signal processing (ISP) chains,providing video data in the standard red-green-blue (“RGB”) color spacefor example (cameras-on-chip).

The control unit for an endoscope or image sensor may be locatedremotely from the image sensor and may be a significant physicaldistance away from the image sensor. When the control unit is remotefrom the sensor, it may be desirable to transmit the data in the digitaldomain because it is largely immune to interference noise and signaldegradation when compared to transmitting an analog data stream. It willbe appreciated that various electrical digital signaling standards maybe used, such as LVDS (low voltage differential signaling), sub-LVDS,SLVS (scalable low voltage signaling) or other electrical digitalsignaling standards.

There may be a strong desire to minimize the number of electricalconductors to reduce the number of pads consuming space on the sensor,and to reduce the complexity and cost of sensor manufacture. Althoughthe addition of analog to digital conversion to the sensor may beadvantageous, the additional area occupied by the conversion circuits isoffset because of the significant reduction in the analog bufferingpower needed due to the early conversion to a digital signal.

In terms of area consumption, given the typical feature size availablein CMOS image sensor (CIS) technologies, it may be preferable in someimplementations to have all the internal logic signals generated on thesame chip as the pixel array via a set of control registers and a simplecommand interface.

Some implementations of the disclosure may include aspects of a combinedsensor and system design that allows for high definition imaging withreduced pixel counts in a highly controlled illumination environment.This may be accomplished by virtue of frame-by-frame pulsing of asingle-color wavelength and switching or alternating each frame betweena single, different color wavelength using a controlled light source inconjunction with high frame capture rates and a specially designedcorresponding monochromatic sensor. Additionally, electromagneticradiation outside the visible light spectrum may be pulsed to enable thegeneration of a hyperspectral image. The pixels may be color agnosticsuch that each pixel may generate data for each pulse of electromagneticradiation, including pulses for red, green, and blue visible lightwavelengths along with other wavelengths that may be used forhyperspectral imaging.

As used herein, monochromatic sensor refers to an unfiltered imagingsensor. Since the pixels are color agnostic, the effective spatialresolution is appreciably higher than for their color (typicallyBayer-pattern filtered) counterparts in conventional single-sensorcameras. They may also have higher quantum efficiency since far fewerincident photons are wasted between the individual pixels. Moreover,Bayer based spatial color modulation requires that the modulationtransfer function (MTF) of the accompanying optics be lowered comparedwith the monochrome modulation, to blur out the color artifactsassociated with the Bayer pattern. This has a detrimental impact on theactual spatial resolution that can be realized with color sensors.

The disclosure is also concerned with a system solution for endoscopyapplications in which the image sensor is resident at the distal end ofthe endoscope. In striving for a minimal area sensor-based system, thereare other design aspects that can be developed, beyond reduction inpixel count. The area of the digital portion of the chip may beminimized In addition, the number of connections to the chip (pads) mayalso be minimized The disclosure describes novel methods that accomplishthese goals for the realization of such a system. This involves thedesign of a full-custom CMOS image sensor with several novel features.

For the purposes of promoting an understanding of the principles inaccordance with the disclosure, reference will now be made to theembodiments illustrated in the drawings and specific language will beused to describe the same. It will nevertheless be understood that nolimitation of the scope of the disclosure is thereby intended. Anyalterations and further modifications of the inventive featuresillustrated herein, and any additional applications of the principles ofthe disclosure as illustrated herein, which would normally occur to oneskilled in the relevant art and having possession of this disclosure,are to be considered within the scope of the disclosure claimed.

Before the structure, systems and methods for producing an image in alight deficient environment are disclosed and described, it is to beunderstood that this disclosure is not limited to the particularstructures, configurations, process steps, and materials disclosedherein as such structures, configurations, process steps, and materialsmay vary somewhat. It is also to be understood that the terminologyemployed herein is used for the purpose of describing particularembodiments only and is not intended to be limiting since the scope ofthe disclosure will be limited only by the appended claims andequivalents thereof.

In describing and claiming the subject matter of the disclosure, thefollowing terminology will be used in accordance with the definitionsset out below.

It must be noted that, as used in this specification and the appendedclaims, the singular forms “a,” “an,” and “the” include plural referentsunless the context clearly dictates otherwise.

As used herein, the terms “comprising,” “including,” “containing,”“characterized by,” and grammatical equivalents thereof are inclusive oropen-ended terms that do not exclude additional, unrecited elements ormethod steps.

As used herein, the phrase “consisting of” and grammatical equivalentsthereof exclude any element or step not specified in the claim.

As used herein, the phrase “consisting essentially of” and grammaticalequivalents thereof limit the scope of a claim to the specifiedmaterials or steps and those that do not materially affect the basic andnovel characteristic or characteristics of the claimed disclosure.

As used herein, the term “proximal” shall refer broadly to the conceptof a portion nearest an origin.

As used herein, the term “distal” shall generally refer to the oppositeof proximal, and thus to the concept of a portion farther from anorigin, or a furthest portion, depending upon the context.

As used herein, color sensors or multi spectrum sensors are thosesensors known to have a color filter array (CFA) thereon to filter theincoming electromagnetic radiation into its separate components. In thevisual range of the electromagnetic spectrum, such a CFA may be built ona Bayer pattern or modification thereon to separate green, red and bluespectrum components of the light.

Referring now to FIGS. 1-5, the systems and methods for producing animage in a light deficient environment will now be described. FIG. 1illustrates a schematic view of a paired sensor and an electromagneticemitter in operation for use in producing an image in a light deficientenvironment. Such a configuration allows for increased functionality ina light controlled or ambient light deficient environment.

It should be noted that as used herein the term “light” is both aparticle and a wavelength and is intended to denote electromagneticradiation that is detectable by a pixel array and may includewavelengths from the visible and non-visible spectrums ofelectromagnetic radiation. The term “partition” is used herein to mean apre-determined range of wavelengths of the electromagnetic spectrum thatis less than the entire spectrum, or in other words, wavelengths thatmake up some portion of the electromagnetic spectrum. As used herein, anemitter is a light source that may be controllable as to the portion ofthe electromagnetic spectrum that is emitted or that may operate as tothe physics of its components, the intensity of the emissions, or theduration of the emission, or all the above. An emitter may emit light inany dithered, diffused, or collimated emission and may be controlleddigitally or through analog methods or systems. As used herein, anelectromagnetic emitter is a source of a burst of electromagnetic energyand includes light sources, such as lasers, LEDs, incandescent light, orany light source that can be digitally controlled.

A pixel array of an image sensor may be paired with an emitterelectronically, such that they are synced during operation for bothreceiving the emissions and for the adjustments made within the system.As can be seen in FIG. 1, an emitter 100 may be tuned to emitelectromagnetic radiation in the form of a laser, which may be pulsed toilluminate an object 110. The emitter 100 may pulse at an interval thatcorresponds to the operation and functionality of a pixel array 122. Theemitter 100 may pulse light in a plurality of electromagnetic partitions105, such that the pixel array receives electromagnetic energy andproduces a data set that corresponds (in time) with each specificelectromagnetic partition 105. For example, FIG. 1 illustrates a systemhaving a monochromatic sensor 120 having a pixel array (black and white)122 and supporting circuitry, which pixel array 122 is sensitive toelectromagnetic radiation of any wavelength. The light emitter 100illustrated in the figure may be a laser emitter that is capable ofemitting a red electromagnetic partition 105 a, a blue electromagneticpartition 105 b, and a green electromagnetic partition 105 c in anydesired sequence. In an embodiment where a hyperspectral image may begenerated, the light emitter 100 may pulse electromagnetic radiation atany wavelength in the electromagnetic spectrum such that a hyperspectralimage may be generated. It will be appreciated that other light emitters100 may be used in FIG. 1 without departing from the scope of thedisclosure, such as digital or analog based emitters.

During operation, the data created by the monochromatic sensor 120 forany individual pulse may be assigned a specific color or wavelengthpartition, wherein the assignment is based on the timing of the pulsedcolor or wavelength partition from the emitter 100. Even though thepixels 122 are not color-dedicated, they can be assigned a color for anygiven data set based on a priori information about the emitter.

In an exemplary embodiment of the disclosure, the emitter 100 pulseselectromagnetic radiation at specialized wavelengths. Such pulses mayenable the generation of a specialized fluorescence image that isparticularly suited for certain medical or diagnostic applications. Inthe exemplary embodiment, at least a portion of the electromagneticradiation emitted by the emitter 100 includes an excitation wavelengthof electromagnetic radiation between 770 nm and 790 nm and between 795nm and 815 nm that cause one or more reagents to fluoresce at awavelength that is different from the excitation wavelength of theportion of the electromagnetic radiation.

In one embodiment, three data sets representing RED, GREEN and BLUEelectromagnetic pulses may be combined to form a single image frame. Oneor more additional data sets representing other wavelength partitionsmay be overlaid on the single image frame that is based on the RED,GREEN, and BLUE pulses. The one or more additional data sets mayrepresent, for example, fluorescence imaging responsive to theexcitation wavelength between 770 nm and 790 nm and between 795 nm and815 nm. The one or more additional data sets may represent fluorescenceimaging and/or hyperspectral that may be overlaid on the single imageframe that is based on the RED, GREEN, and BLUE pulses.

It will be appreciated that the disclosure is not limited to anyparticular color combination or any particular electromagneticpartition, and that any color combination or any electromagneticpartition may be used in place of RED, GREEN and BLUE, such as Cyan,Magenta and Yellow; Ultraviolet; infra-red; any combination of theforegoing, or any other color combination, including all visible andnon-visible wavelengths, without departing from the scope of thedisclosure. In the figure, the object 110 to be imaged contains a redportion 110 a, green portion 110 b and a blue portion 110 c. Asillustrated in the figure, the reflected light from the electromagneticpulses only contains the data for the portion of the object having thespecific color that corresponds to the pulsed color partition. Thoseseparate color (or color interval) data sets can then be used toreconstruct the image by combining the data sets at 130.

In one embodiment, a plurality of data sets representing RED, GREEN, andBLUE electromagnetic pulses along with additional wavelength partitionsalong the electromagnetic spectrum may be combined to form a singleimage frame having an RGB image with hyperspectral image data overlaidon the RGB image. Depending on the application or instance, differentcombinations of wavelength data sets may be desirable. For example, insome implementations, a data set representing specific wavelengthpartitions may be used to generate a specialized hyperspectral image fordiagnosing a particular medical condition, investigating certain bodytissues, and so forth.

As illustrated in FIG. 2, implementations of the present disclosure maycomprise or utilize a special purpose or general-purpose computer,including computer hardware, such as, for example, one or moreprocessors and system memory, as discussed in greater detail below.Implementations within the scope of the present disclosure may alsoinclude physical and other computer-readable media for carrying orstoring computer-executable instructions and/or data structures. Suchcomputer-readable media can be any available media that can be accessedby a general purpose or special purpose computer system.Computer-readable media that store computer-executable instructions arecomputer storage media (devices). Computer-readable media that carrycomputer-executable instructions are transmission media. Thus, by way ofexample, and not limitation, implementations of the disclosure cancomprise at least two distinctly different kinds of computer-readablemedia: computer storage media (devices) and transmission media.

Computer storage media (devices) includes RAM, ROM, EEPROM, CD-ROM,solid state drives (“SSDs”) (e.g., based on RAM), Flash memory,phase-change memory (“PCM”), other types of memory, other optical diskstorage, magnetic disk storage or other magnetic storage devices, or anyother medium which can be used to store desired program code means inthe form of computer-executable instructions or data structures andwhich can be accessed by a general purpose or special purpose computer.

A “network” is defined as one or more data links that enable thetransport of electronic data between computer systems and/or modulesand/or other electronic devices. In an implementation, a sensor andcamera control unit may be networked to communicate with each other, andother components, connected over the network to which they areconnected. When information is transferred or provided over a network oranother communications connection (either hardwired, wireless, or acombination of hardwired or wireless) to a computer, the computerproperly views the connection as a transmission medium. Transmissionsmedia can include a network and/or data links, which can be used tocarry desired program code means in the form of computer-executableinstructions or data structures and which can be accessed by a generalpurpose or special purpose computer. Combinations of the above shouldalso be included within the scope of computer-readable media.

Further, upon reaching various computer system components, program codemeans in the form of computer-executable instructions or data structuresthat can be transferred automatically from transmission media tocomputer storage media (devices) (or vice versa). For example,computer-executable instructions or data structures received over anetwork or data link can be buffered in RAM within a network interfacemodule (e.g., a “NIC”), and then eventually transferred to computersystem RAM and/or to less volatile computer storage media (devices) at acomputer system. RAM can also include solid state drives (SSDs or PCIxbased real time memory tiered storage, such as FusionIO). Thus, itshould be understood that computer storage media (devices) can beincluded in computer system components that also (or even primarily)utilize transmission media.

Computer-executable instructions comprise, for example, instructions anddata which, when executed at a processor, cause a general-purposecomputer, special purpose computer, or special purpose processing deviceto perform a certain function or group of functions. The computerexecutable instructions may be, for example, binaries, intermediateformat instructions such as assembly language, or even source code.Although the subject matter has been described in language specific tostructural features and/or methodological acts, it is to be understoodthat the subject matter defined in the appended claims is notnecessarily limited to the described features or acts described above.Rather, the described features and acts are disclosed as example formsof implementing the claims.

Those skilled in the art will appreciate that the disclosure may bepracticed in network computing environments with many types of computersystem configurations, including, personal computers, desktop computers,laptop computers, message processors, control units, camera controlunits, hand-held devices, hand pieces, multi-processor systems,microprocessor-based or programmable consumer electronics, network PCs,minicomputers, mainframe computers, mobile telephones, PDAs, tablets,pagers, routers, switches, various storage devices, and the like. Itshould be noted that any of the above-mentioned computing devices may beprovided by or located within a brick and mortar location. Thedisclosure may also be practiced in distributed system environmentswhere local and remote computer systems, which are linked (either byhardwired data links, wireless data links, or by a combination ofhardwired and wireless data links) through a network, both performtasks. In a distributed system environment, program modules may belocated in both local and remote memory storage devices.

Further, where appropriate, functions described herein can be performedin one or more of: hardware, software, firmware, digital components, oranalog components. For example, one or more application specificintegrated circuits (ASICs) or field programmable gate arrays (FPGAs)can be programmed to carry out one or more of the systems and proceduresdescribed herein. Certain terms are used throughout the followingdescription and Claims to refer to particular system components. As oneskilled in the art will appreciate, components may be referred to bydifferent names This document does not intend to distinguish betweencomponents that differ in name, but not function.

FIG. 2 is a block diagram illustrating an example computing device 150.Computing device 150 may be used to perform various procedures, such asthose discussed herein. Computing device 150 can function as a server, aclient, or any other computing entity. Computing device 150 can performvarious monitoring functions as discussed herein, and can execute one ormore application programs, such as the application programs describedherein. Computing device 150 can be any of a wide variety of computingdevices, such as a desktop computer, a notebook computer, a servercomputer, a handheld computer, camera control unit, tablet computer andthe like.

Computing device 150 includes one or more processor(s) 152, one or morememory device(s) 154, one or more interface(s) 156, one or more massstorage device(s) 158, one or more Input/Output (I/O) device(s) 160, anda display device 180 all of which are coupled to a bus 162. Processor(s)152 include one or more processors or controllers that executeinstructions stored in memory device(s) 154 and/or mass storagedevice(s) 158. Processor(s) 152 may also include various types ofcomputer-readable media, such as cache memory.

Memory device(s) 154 include various computer-readable media, such asvolatile memory (e.g., random access memory (RAM) 164) and/ornonvolatile memory (e.g., read-only memory (ROM) 166). Memory device(s)154 may also include rewritable ROM, such as Flash memory.

Mass storage device(s) 158 include various computer readable media, suchas magnetic tapes, magnetic disks, optical disks, solid-state memory(e.g., Flash memory), and so forth. As shown in FIG. 2, a particularmass storage device is a hard disk drive 174. Various drives may also beincluded in mass storage device(s) 158 to enable reading from and/orwriting to the various computer readable media. Mass storage device(s)158 include removable media 176 and/or non-removable media.

I/O device(s) 160 include various devices that allow data and/or otherinformation to be input to or retrieved from computing device 150.Example I/O device(s) 160 include digital imaging devices,electromagnetic sensors and emitters, cursor control devices, keyboards,keypads, microphones, monitors or other display devices, speakers,printers, network interface cards, modems, lenses, CCDs or other imagecapture devices, and the like.

Display device 180 includes any type of device capable of displayinginformation to one or more users of computing device 150. Examples ofdisplay device 180 include a monitor, display terminal, video projectiondevice, and the like.

Interface(s) 106 include various interfaces that allow computing device150 to interact with other systems, devices, or computing environments.Example interface(s) 156 may include any number of different networkinterfaces 170, such as interfaces to local area networks (LANs), widearea networks (WANs), wireless networks, and the Internet. Otherinterface(s) include user interface 168 and peripheral device interface172. The interface(s) 156 may also include one or more user interfaceelements 168. The interface(s) 156 may also include one or moreperipheral interfaces such as interfaces for printers, pointing devices(mice, track pad, etc.), keyboards, and the like.

Bus 162 allows processor(s) 152, memory device(s) 154, interface(s) 156,mass storage device(s) 158, and I/O device(s) 160 to communicate withone another, as well as other devices or components coupled to bus 162.Bus 162 represents one or more of several types of bus structures, suchas a system bus, PCI bus, IEEE 1394 bus, USB bus, and so forth.

For purposes of illustration, programs and other executable programcomponents are shown herein as discrete blocks, although it isunderstood that such programs and components may reside at various timesin different storage components of computing device 150, and areexecuted by processor(s) 152. Alternatively, the systems and proceduresdescribed herein can be implemented in hardware, or a combination ofhardware, software, and/or firmware. For example, one or moreapplication specific integrated circuits (ASICs) or field programmablegate arrays (FPGAs) can be programmed to carry out one or more of thesystems and procedures described herein.

FIG. 2A illustrates the operational cycles of a sensor used in rollingreadout mode or during the sensor readout 200. The frame readout maystart at and may be represented by vertical line 210. The read-outperiod is represented by the diagonal or slanted line 202. The sensormay be read out on a row by row basis, the top of the downwards slantededge being the sensor top row 212 and the bottom of the downwardsslanted edge being the sensor bottom row 214. The time between the lastrow readout and the next readout cycle may be called the blanking time216. It should be noted that some of the sensor pixel rows might becovered with a light shield (e.g., a metal coating or any othersubstantially black layer of another material type). These covered pixelrows may be referred to as optical black rows 218 and 220. Optical blackrows 218 and 220 may be used as input for correction algorithms As shownin FIG. 2A, these optical black rows 218 and 220 may be located on thetop of the pixel array or at the bottom of the pixel array or at the topand the bottom of the pixel array. FIG. 2B illustrates a process ofcontrolling the amount of electromagnetic radiation, e.g., light, thatis exposed to a pixel, thereby integrated or accumulated by the pixel.It will be appreciated that photons are elementary particles ofelectromagnetic radiation. Photons are integrated, absorbed, oraccumulated by each pixel and converted into an electrical charge orcurrent. An electronic shutter or rolling shutter (shown by dashed line222) may be used to start the integration time by resetting the pixel.The light will then integrate until the next readout phase. The positionof the electronic shutter 222 can be moved between two readout cycles202 to control the pixel saturation for a given amount of light. Itshould be noted that this technique allows for a constant integrationtime between two different lines but introduces a delay when moving fromtop to bottom rows. FIG. 2C illustrates the case where the electronicshutter 222 has been removed. In this configuration, the integration ofthe incoming light may start during readout 202 and may end at the nextreadout cycle 202, which also defines the start of the next integration.FIG. 2D shows a configuration without an electronic shutter 222, butwith a controlled and pulsed light 230 during the blanking time 216.This ensures that all rows see the same light issued from the same lightpulse 230. In other words, each row will start its integration in a darkenvironment, which may be at the optical black back row 220 of read outframe (m) for a maximum light pulse width, and will then receive a lightstrobe and will end its integration in a dark environment, which may beat the optical black front row 218 of the next succeeding read out frame(m+1) for a maximum light pulse width. In the FIG. 2D example, the imagegenerated from the light pulse will be solely available during frame(m+1) readout without any interference with frames (m) and (m+2). Itshould be noted that the condition to have a light pulse to be read outonly in one frame and not interfere with neighboring frames is to havethe given light pulse firing during the blanking time 216. Because theoptical black rows 218, 220 are insensitive to light, the optical blackback rows 220 time of frame (m) and the optical black front rows 218time of frame (m+1) can be added to the blanking time 216 to determinethe maximum range of the firing time of the light pulse 230. Asillustrated in the FIG. 2A, a sensor may be cycled many times to receivedata for each pulsed color or wavelength (e.g., Red, Green, Blue, orother wavelength on the electromagnetic spectrum). Each cycle may betimed. In an embodiment, the cycles may be timed to operate within aninterval of 16.67 ms. In another embodiment, the cycles may be timed tooperate within an interval of 8.3 ms. It will be appreciated that othertiming intervals are contemplated by the disclosure and are intended tofall within the scope of this disclosure.

FIG. 3 graphically illustrates the operation of an embodiment of anelectromagnetic emitter. An emitter may be timed to correspond with thecycles of a sensor, such that electromagnetic radiation is emittedwithin the sensor operation cycle and/or during a portion of the sensoroperation cycle. FIG. 3 illustrates Pulse 1 at 302, Pulse 2 at 304, andPulse 3 at 306. In an embodiment, the emitter may pulse during theread-out portion 202 of the sensor operation cycle. In an embodiment,the emitter may pulse during the blanking portion 216 of the sensoroperation cycle. In an embodiment, the emitter may pulse for a durationthat is during portions of two or more sensor operational cycles. In anembodiment, the emitter may begin a pulse during the blanking portion216, or during the optical black portion 220 of the readout portion 202,and end the pulse during the readout portion 202, or during the opticalblack portion 218 of the readout portion 202 of the next succeedingcycle. It will be understood that any combination of the above isintended to fall within the scope of this disclosure as long as thepulse of the emitter and the cycle of the sensor correspond.

FIG. 4 graphically represents varying the duration and magnitude of theemitted electromagnetic pulse (e.g., Pulse 1 at 402, Pulse 2 at 404, andPulse 3 at 406) to control exposure. An emitter having a fixed outputmagnitude may be pulsed during any of the cycles noted above in relationto FIGS. 2D and 3 for an interval to provide the needed electromagneticenergy to the pixel array. An emitter having a fixed output magnitudemay be pulsed at a longer interval of time, thereby providing moreelectromagnetic energy to the pixels or the emitter may be pulsed at ashorter interval of time, thereby providing less electromagnetic energy.Whether a longer or shorter interval time is needed depends upon theoperational conditions.

In contrast to adjusting the interval of time that the emitter pulses afixed output magnitude, the magnitude of the emission itself may beincreased to provide more electromagnetic energy to the pixels.Similarly, decreasing the magnitude of the pulse provides lesselectromagnetic energy to the pixels. It should be noted that anembodiment of the system may have the ability to adjust both magnitudeand duration concurrently, if desired. Additionally, the sensor may beadjusted to increase its sensitivity and duration as desired for optimalimage quality. FIG. 4 illustrates varying the magnitude and duration ofthe pulses. In the illustration, Pulse 1 at 402 has a higher magnitudeor intensity than either Pulse 2 at 404 or Pulse 3 at 406. Additionally,Pulse 1 at 402 has a shorter duration than Pulse 2 at 404 or Pulse 3 at406, such that the electromagnetic energy provided by the pulse isillustrated by the area under the pulse shown in the illustration. Inthe illustration, Pulse 2 at 404 has a relatively low magnitude orintensity and a longer duration when compared to either Pulse 1 at 402or Pulse 3 at 406. Finally, in the illustration, Pulse 3 at 406 has anintermediate magnitude or intensity and duration, when compared to Pulse1 at 402 and Pulse 2 at 404.

FIG. 5 is a graphical representation of an embodiment of the disclosurecombining the operational cycles, the electromagnetic emitter and theemitted electromagnetic pulses of FIGS. 2-4 to demonstrate the imagingsystem during operation in accordance with the principles and teachingsof the disclosure. As can be seen in the figure, the electromagneticemitter pulses the emissions primarily during the blanking period 216 ofthe sensor, such that the pixels will be charged and ready to readduring the read-out portion 202 of the sensor cycle. The dashed lineportions in the pulse (from FIG. 3) illustrate the potential or abilityto emit electromagnetic energy during the optical black portions 220 and218 of the read cycle (sensor cycle) 200 if additional time is needed ordesired to pulse electromagnetic energy.

Referring now to FIGS. 6-9A, FIG. 6 illustrates a schematic of twodistinct processes over a period of time from t(0) to t(1) for recordinga frame of video for full spectrum light and partitioned spectrum light.It should be noted that color sensors have a color filter array (CFA)for filtering out certain wavelengths of light per pixel commonly usedfor full spectrum light reception. An example of a CFA is a Bayerpattern. Because the color sensor may comprise pixels within the arraythat are made sensitive to a single color from within the full spectrum,a reduced resolution image results because the pixel array has pixelspaces dedicated to only a single color of light within the fullspectrum. Usually such an arrangement is formed in a checkerboard typepattern across the entire array.

In contrast, when partitioned spectrums of light are used a sensor canbe made to be sensitive or responsive to the magnitude of all lightenergy because the pixel array will be instructed that it is sensingelectromagnetic energy from a predetermined partition of the fullspectrum of electromagnetic energy in each cycle. Therefore, to form animage the sensor need only be cycled with a plurality of differingpartitions from within the full spectrum of light and then reassemblingthe image to display a predetermined mixture of color values for everypixel across the array. Accordingly, a higher resolution image is alsoprovided because there are reduced distances as compared to a Bayersensor between pixel centers of the same color sensitivity for each ofthe color pulses. As a result, the formed colored image has a highermodulation transfer function (MTF). Because the image from each colorpartition frame cycle, has a higher resolution, the resultant imagecreated when the partitioned light frames are combined into a full colorframe, also has a higher resolution. In other words, because each andevery pixel within the array (instead of, at most, every second pixel ina sensor with color filter) is sensing the magnitudes of energy for agiven pulse and a given scene, just fractions of time apart, a higherresolution image is created for each scene with less derived (lessaccurate) data needing to be introduced.

For example, white or full spectrum visible light is a combination ofred, green and blue light. In the embodiment shown in FIG. 6, it can beseen that in both the partitioned spectrum process 620 and full spectrumprocess 610 the time to capture an image is t(0) to t(1). In the fullspectrum process 610, white light or full spectrum electromagneticenergy is emitted at 612. At 614, the white or full spectrumelectromagnetic energy is sensed. At 616, the image is processed anddisplayed. Thus, between time t(0) and t(1), the image has beenprocessed and displayed. Conversely, in the partitioned spectrum process620, a first partition is emitted at 622 and sensed at 624. At 626, asecond partition is emitted and then sensed at 628. At 630, a thirdpartition is emitted and sensed at 632. At 634, the image is processedand displayed. It will be appreciated that any system using an imagesensor cycle that is at least two times faster than the white lightcycle is intended to fall within the scope of the disclosure.

As can be seen graphically in the embodiment illustrated in FIG. 6between times t(0) and t(1), the sensor for the partitioned spectrumsystem 620 has cycled three times for every one of the full spectrumsystem. In the partitioned spectrum system 620, the first of the threesensor cycles are for a green spectrum 622 and 624, the second of thethree is for a red spectrum 626 and 628, and the third is for a bluespectrum 630 and 632. Thus, in an embodiment, wherein the display device(LCD panel) operates at 50-60 frames per second, a partitioned lightsystem should operate at 150-180 frames per second to maintain thecontinuity and smoothness of the displayed video.

In other embodiments, there may be different capture and display framerates. Furthermore, the average capture rate could be any multiple ofthe display rate.

In an embodiment, it may be desired that not all partitions berepresented equally within the system frame rate. In other words, notall light sources have to be pulsed with the same regularity so as toemphasize and de-emphasize aspects of the recorded scene as desired bythe users. It should also be understood that non-visible and visiblepartitions of the electromagnetic spectrum may be pulsed together withina system with their respective data value being stitched into the videooutput as desired for display to a user.

An embodiment may comprise a pulse cycle pattern as follows:

i. Green pulse;

ii. Red pulse;

iii. Blue pulse;

iv. Green pulse;

v. Red pulse;

vi. Blue pulse;

vii. Infra-red (IR) pulse;

viii. (Repeat)

As can be seen in the example, an infrared partition or a specializedwavelength partition (e.g., 513-545 nm, 565-585 nm, and/or 900-100 nm)may be pulsed at a rate differing from the rates of the other partitionpulses. This may be done to emphasize a certain aspect of the scene,with the IR data simply being overlaid with the other data in the videooutput to make the desired emphasis. It should be noted that theaddition of an electromagnetic partition on top of the RED, GREEN, andBLUE partitions does not necessarily require the serialized system tooperate at four times the rate of a full spectrum non-serial systembecause every partition does not have to be represented equally in thepulse pattern. As seen in the embodiment, the addition of a partitionpulse that is represented less in a pulse pattern (infrared in an aboveexample), would result in an increase of less than 20% of the cyclingspeed of the sensor in order accommodate the irregular partitionsampling.

In an embodiment, an electromagnetic partition may be emitted that issensitive to dyes or materials that are used to highlight aspects of ascene. In the embodiment, it may be sufficient to highlight the locationof the dyes or materials without need for high resolution. In such anembodiment, the dye sensitive electromagnetic partition may be cycledmuch less frequently than the other partitions in the system to includethe emphasized data.

In various embodiments, the pulse cycle pattern may include any of thefollowing wavelengths in any suitable order. Such wavelengths may beparticularly suited for determining multispectral or hyperspectral imagedata or for determining image data based on a fluorescent reagentrelaxation emission:

i. 465±5 nm;

ii. 533±4 nm;

iii. 638±5 nm;

iv. 780±5 nm;

v. 805±5 nm;

vi. 975±5 nm;

vii. 577±2 nm; or

viii. 523±4 nm.

The partition cycles may be divided so as to accommodate or approximatevarious imaging and video standards. In an embodiment, the partitioncycles may comprise pulses of electromagnetic energy in the Red, Green,Blue spectrum as follows as illustrated best in FIGS. 7A-7D. In FIG. 7A,the different light intensities have been achieved by modulating thelight pulse width or duration within the working range shown by thevertical grey dashed lines. In FIG. 7B, the different light intensitieshave been achieved by modulating the light power or the power of theelectromagnetic emitter, which may be a laser or LED emitter, butkeeping the pulse width or duration constant. FIG. 7C shows the casewhere both the light power and the light pulse width are beingmodulated, leading to greater flexibility. The partition cycles may useCMY, IR and ultraviolet using a non-visible pulse source mixed withvisible pulse sources and any other color space required to produce animage or approximate a desired video standard that is currently known oryet to be developed. It should also be understood that a system may beable to switch between the color spaces on the fly to provide thedesired image output quality.

In an embodiment using color spaces Green-Blue-Green-Red (as seen inFIG. 7D) it may be desirous to pulse the luminance components more oftenthan the chrominance components because users are generally moresensitive to light magnitude differences than to light colordifferences. This principle can be exploited using a mono-chromaticsensor as illustrated in FIG. 7D. In FIG. 7D, green, which contains themost luminance information, may be pulsed more often or with moreintensity in a (G-B-G-R-G-B-G-R. . . ) scheme to obtain the luminancedata. Such a configuration would create a video stream that hasperceptively more detail, without creating and transmittingunperceivable data.

In an embodiment, duplicating the pulse of a weaker partition may beused to produce an output that has been adjusted for the weaker pulse.For example, blue laser light is considered weak relative to thesensitivity of silicon-based pixels and is difficult to produce incomparison to the red or green light, and therefore may be pulsed moreoften during a frame cycle to compensate for the weakness of the light.These additional pulses may be done serially over time or by usingmultiple lasers that simultaneously pulse to produce the desiredcompensation effect. It should be noted that by pulsing during ablanking period (time during which the sensor is not reading out thepixel array), the sensor is insensitive to differences/mismatchesbetween lasers of the same kind and simply accumulates the light for thedesired output. In another embodiment, the maximum light pulse range maybe different from frame to frame. This is shown in FIG. 7E where thelight pulses are different from frame to frame. The sensor may be builtto be able to program different blanking times with a repeating patternof 2 or 3 or 4 or n frames. In FIG. 7E, 4 different light pulses areillustrated, and Pulse 1 may repeat for example after Pulse 4 and mayhave a pattern of 4 frames with different blanking times. This techniquecan be used to place the most powerful partition on the smallestblanking time and therefore allow the weakest partition to have widerpulse on one of the next frames without the need of increasing thereadout speed. The reconstructed frame can still have a regular patternfrom frame to frame as it is constituted of many pulsed frames.

As can be seen in FIG. 8, because each partitioned spectrum of light mayhave different energy values, the sensor and/or light emitter may beadjusted to compensate for the differences in the energy values. At 810,the data obtained from the histogram from a previous frame may beanalyzed. At 820, the sensor may be adjusted as noted below.Additionally, at 830, the emitter may be adjusted. At 840, the image maybe obtained from the adjusted sample time from the sensor or the imagemay be obtained with adjusted (either increased or decreased) emittedlight, or a combination of the above. For example, because the red lightspectrum is more readily detected by a sensor within the system than theblue light spectrum, the sensor can be adjusted to be less sensitiveduring the red partition cycle and more sensitive during the bluepartition cycle because of the low Quantum Efficiency that the bluepartition has with respect to silicon (illustrated best in FIG. 9).Similarly, the emitter may be adjusted to provide an adjusted partition(e.g., higher or lower intensity and duration). Further, adjustments maybe made at the sensor and emitter level both. The emitter may also bedesigned to emit at one specific frequency or may be changed to emitmultiple frequencies of a specific partition to broaden the spectrum oflight being emitted, if desired for a particular application.

FIG. 10 shows a schematic of an unshared 4T pixel. The TX signal is usedto transfer accumulated charges from the photo diode (PPD) to thefloating diffusion (FD). The reset signal is used to reset the FD to thereset bus. If reset and TX signals are “On” at the same time, the PPD isconstantly reset (each photo charge generated in the PPD is directlycollected at the reset bus) and the PPD is always empty. Usual pixelarray implementation includes a horizontal reset line that attaches thereset signals of all pixels within one row and a horizontal TX line thatattaches the TX signals of all pixels within one row.

In an embodiment, timing of the sensor sensibility adjustment isillustrated, and sensor sensibility adjustment can be achieved using aglobal reset mechanism (i.e., a means of firing all pixel array resetsignals at once) and a global TX mechanism (i.e., a means of firing allpixel array TX signals at once). This is shown in FIG. 11. In this case,the light pulse is constant in duration and amplitude, but the lightintegrated in all pixels starts with the “on” to “off” transition of theglobal TX and ends with the light pulse. Therefore, the modulation isachieved by moving the falling edge of the global TX pulse.

Conversely, the emitter may emit red light at a lesser intensity thanblue light to produce a correctly exposed image (illustrated best inFIG. 12). At 1210, the data obtained from the histogram from a previousframe may be analyzed. At 1220, the emitter may be adjusted. At 1230,the image may be obtained from the adjusted emitted light. Additionally,in an embodiment both the emitter and the sensor can be adjustedconcurrently.

Reconstructing the partitioned spectrum frames into a full spectrumframe for later output could be as simple as blending the sensed valuesfor each pixel in the array in some embodiments. Additionally, theblending and mixing of values may be simple averages or may be tuned toa predetermined lookup table (LUT) of values for desired outputs. In anembodiment of a system using partitioned light spectrums, the sensedvalues may be post-processed or further refined remotely from the sensorby an image or secondary processor, and just before being output to adisplay.

FIG. 13 illustrates a basic example at 1300 of a monochrome ISP and howan ISP chain may be assembled for the purpose of generating sRGB imagesequences from raw sensor data, yielded in the presence of the G-R-G-Blight pulsing scheme.

The first stage is concerned with making corrections (see 1302, 1304 and1306 in FIG. 13) to account for any non-idealities in the sensortechnology for which it is most appropriate to work in the raw datadomain (see FIG. 21).

At the next stage, two frames (see 1308 and 1310 in FIG. 13) would bebuffered since each final frame derives data from three raw frames. Theframe reconstruction at 1314 would proceed by sampling data from thecurrent frame and the two buffered frames (1308 and/or 1310). Thereconstruction process results in full color frames in linear RGB colorspace.

In this example, the white balance coefficients at 1318 and colorcorrection matrix at 1320 are applied before converting to YCbCr spaceat 1322 for subsequent edge enhancement at 1324. After edge enhancementat 1324, images are transformed back to linear RGB at 1326 for scalingat 1328, if applicable.

Finally, the gamma transfer function at 1330 would be applied totranslate the data into the sRGB domain at 1332.

FIG. 14 is an embodiment example of color fusion hardware. The colorfusion hardware takes in an RGBGRGBGRGBG video data stream at 1402 andconverts it to a parallel RGB video data stream at 1405. The bit widthon the input side may be, e.g., 12 bits per color. The output width forthat example would be 36 bits per pixel. Other embodiments may havedifferent initial bit widths and 3 times that number for the outputwidth. The memory writer block takes as its input the RGBG video streamat 1402 and writes each frame to its correct frame memory buffer at 1404(the memory writer triggers off the same pulse generator 1410 that runsthe laser light source). As illustrated at 1404, writing to the memoryfollows the pattern, Red, Green 1, Blue, Green 2, and then starts backwith Red again. At 1406, the memory reader reads three frames at once toconstruct an RGB pixel. Each pixel is three times the bit width of anindividual color component. The reader also triggers off the laser pulsegenerator at 1410. The reader waits until Red, Green 1 and Blue frameshave been written, then proceeds to read them out in parallel while thewriter continues writing Green 2 and starts back on Red. When Redcompletes the reader begins reading from Blue, Green 2 and Red. Thispattern continues indefinitely.

Referring now to FIGS. 15 and 16, the RG1BG2RG1BG2 patternreconstruction illustrated in FIG. 16 allows 60 fps output with 120 fpsinput in an embodiment. Each consecutive frame contains either a red orblue component from the previous frame. In FIG. 16, each color componentis available in 8.3 ms and the resulting reconstructed frame has aperiod of 16.67 ms. In general, for this pulsing scheme, thereconstructed frame has a period twice of that of the incoming coloredframe as shown in FIG. 15. In other embodiments, different pulsingschemes may be employed. For example, embodiments may be based on thetiming of each color component or frame (T1) and the reconstructed framehaving a period twice that of the incoming color frame (2×T1). Differentframes within the sequence may have different frame periods and theaverage capture rate could be any multiple of the final frame rate.

FIGS. 17-20 illustrate color correction methods and hardware schematicsfor use with a partitioned light system. It is common in digital imagingto manipulate the values within image data to correct the output to meetuser expectations or to highlight certain aspects of the imaged object.Most commonly this is done in satellite images that are tuned andadjusted to emphasize one data type over another. Most often, insatellite acquired data there is the full spectrum of electromagneticenergy available because the light source is not controlled, i.e., thesun is the light source. In contrast, there are imaging conditions wherethe light is controlled and even provided by a user. In such situations,calibration of the image data is still desirable, because withoutcalibration improper emphasis may be given to certain data over otherdata. In a system where the light is controlled by the user, it isadvantageous to provide emissions of light that are known to the userand may be only a portion of the electromagnetic spectrum or a pluralityof portions of the full electromagnetic spectrum. Calibration remainsimportant to meet the expectations of the users and check for faultswithin the system. One method of calibration can be a table of expectedvalues for a given imaging condition that can be compared to the datafrom the sensor. An embodiment may include a color neutral scene havingknown values that should be output by the imaging device and the devicemay be adjusted to meet those known values when the device samples thecolor neutral scene.

In use, and upon start up, the system may sample a color neutral sceneat 1710 (as illustrated in FIG. 17) by running a full cycle of aplurality of electromagnetic spectrum partitions at 1702. A table ofvalues 1708 may be formed to produce a histogram for the frame at 1704.The values of the frame can be compared to the known or expected valuesfrom the color neutral scene at 1706. The imaging device may then beadjusted to meet the desired output at 1712. In an embodimentillustrated in FIG. 17, the system may comprise an image signalprocessor (ISP) that may be adjusted to color correct the imagingdevice.

It should be noted that because each partitioned spectrum of light mayhave different energy values, the sensor and/or light emitter may beadjusted to compensate for the differences in the energy values. Forexample, in an embodiment, because the blue light spectrum has a lowerquantum efficiency than the red light spectrum with regard to siliconbased imagers, the sensor's responsiveness can then be adjusted to beless responsive during the red cycle and more responsive during the bluecycle. Conversely, the emitter may emit blue light at a higherintensity, because of the lower quantum efficiency of the blue light,than red light to produce a correctly exposed image.

In an embodiment illustrated in FIG. 18, where the light sourceemissions are provided and controllable by the system, adjustment ofthose light emissions can be made to color correct an image at 1800.Adjustments may be made to any aspect of the emitted light such asmagnitude, duration (i.e., time-on), or the range within the spectrumpartition. Additionally, both the emitter and the sensor can be adjustedconcurrently in some embodiments as shown in FIG. 19.

To reduce the amount of noise and artifacts within the outputted imagestream or video, fractionalized adjustments may be made to the sensor oremitter within the system as can be seen in FIG. 20. Illustrated in FIG.20 is a system 2000 where both the emitter 2006 and the sensor 2008 canbe adjusted, but an imaging device where either the emitter or sensor isadjusted during use or for a portion of use is also contemplated and iswithin the scope of this disclosure. It may be advantageous to adjustonly the emitter during one portion of use and adjust only the sensorduring another portion of use, while further yet adjusting bothconcurrently during a portion of use. In any of the above embodiments,improved image quality may be obtained by limiting the overalladjustments that the system can make between frame cycles. In otherwords, an embodiment may be limited such that the emitter may only beadjusted a fraction of its operational range at any time between frames.Likewise, the sensor may be limited such that it may only be adjusted afraction of its operational range at any time between frames.Furthermore, both the emitter and sensor may be limited such that theymay only be adjusted together at a fraction of their respectiveoperational ranges at any time between frames in an embodiment.

In an exemplary embodiment, a fractional adjustment of the componentswithin the system may be performed, for example, at about 0.1 dB of theoperational range of the components to correct the exposure of theprevious frame. The 0.1 dB is merely an example and it should be notedthat is other embodiments the allowed adjustment of the components maybe any portion of their respective operational ranges. The components ofthe system can change by intensity or duration adjustment that isgenerally governed by the number of bits (resolution) output by thecomponent. The component resolution may be typically between a range ofabout 10-24 bits but should not be limited to this range as it isintended to include resolutions for components that are yet to bedeveloped in addition to those that are currently available. Forexample, after a first frame it is determined that the scene is too bluewhen observed, then the emitter may be adjusted to decrease themagnitude or duration of the pulse of the blue light during the bluecycle of the system by a fractional adjustment as discussed above, suchas about 0.1 dB.

In this exemplary embodiment, more than 10 percent may have been needed,but the system has limited itself to 0.1 dB adjustment of theoperational range per system cycle. Accordingly, during the next systemcycle the blue light can then be adjusted again, if needed.Fractionalized adjustment between cycles may have a damping effect ofthe outputted imaged and will reduce the noise and artifacts whenoperating emitters and sensors at their operation extremes. It may bedetermined that any fractional amount of the components' operationalrange of adjustment may be used as a limiting factor, or it may bedetermined that certain embodiments of the system may comprisecomponents that may be adjusted over their entire operational range.

Additionally, the optical black area of any image sensor may be used toaid in image correction and noise reduction. In an embodiment, thevalues read from the optical black area may be compared to those of theactive pixel region of a sensor to establish a reference point to beused in image data processing. FIG. 21 shows the kind of sensorcorrection processes that might be employed in a color pulsed system.CMOS image sensors typically have multiple non-idealities that have adetrimental effect on image quality, particularly in low light. Chiefamong these are fixed pattern noise and line noise. Fixed pattern noiseis a dispersion in the offsets of the sense elements. Typically, most ofthe FPN is a pixel to pixel dispersion which stems, among other sources,from random variations in dark current from photodiode to photodiode.This looks very unnatural to the viewer. Even more egregious is columnFPN, resulting from offsets in the readout chain associated with aparticular columns of pixels. This results in perceived vertical stripeswithin the image.

Being in total control of the illumination has the benefit that entireframes of dark data may periodically be acquired and used to correct forthe pixel and column offsets. In the illustrated example, a single framebuffer may be used to make a running average of the whole frame withoutlight using, e.g., simple exponential smoothing. This dark average framewould be subtracted from every illuminated frame during regularoperation.

Line-Noise is a stochastic temporal variation in the offsets of pixelswithin each row. Since it is temporal, the correction must be computedanew for each line and each frame. For this purpose, there are usuallymany optically blind (OB) pixels within each row in the array, whichmust first be sampled to assess the line offset before sampling thelight sensitive pixels. The line offset is then simply subtracted duringthe line noise correction process.

In the example in FIG. 21, there are other corrections concerned withgetting the data into the proper order, monitoring and controlling thevoltage offset in the analog domain (black clamp) andidentifying/correcting individual defective pixels.

FIGS. 22 and 23 illustrate method and hardware schematics for increasingthe dynamic range within a closed or limited light environment. In anembodiment, exposure inputs may be input at different levels over timeand combine to produce greater dynamic range. As can be seen in FIG. 22,an imaging system may be cycled at a first intensity for a first cycleat 2202 and then subsequently cycled at a second intensity for a secondcycle at 2204, and then by combining those first and second cycles intoa single frame at 2206 so that greater dynamic range can be achieved.Greater dynamic range may be especially desirable because of the limitedspace environment in which an imaging device is used. In limited spaceenvironments that are light deficient or dark, except for the lightprovided by the light source, and where the light source is close to thelight emitter, exposure has an exponential relationship to distance. Forexample, objects near the light source and optical opening of theimaging device tend to be over exposed, while objects farther away tendto be extremely under exposed because there is very little (in any)ambient light present.

As can be seen in FIG. 23, the cycles of a system having emissions ofelectromagnetic energy in a plurality of partitions may be seriallycycled according to the partitions of electromagnetic spectrum at 2300.For example, in an embodiment where the emitter emits lasers in adistinct red partition, a distinct blue partition, and a distinct greenpartition, the two cycle data sets that are going to be combined may bein the form of:

i. red at intensity one at 2302,

ii. red at intensity two at 2304,

iii. blue at intensity one at 2302,

iv. blue at intensity two at 2304,

v. green at intensity one at 2302,

vi. green at intensity two at 2304.

Alternatively, the system may be cycled in the form of:

i. red at intensity one at 2302,

ii. blue at intensity one at 2302,

iii. green at intensity one at 2302,

iv. red at intensity two at 2304,

v. blue at intensity two at 2304,

vi. green at intensity two at 2304.

In such an embodiment, a first image may be derived from the intensityone values, and a second image may be derived from the intensity twovalues, and then combined or processed as complete image data sets at2310 rather than their component parts.

It is contemplated to be within the scope of this disclosure that anynumber of emission partitions may be used in any order. As seen in FIG.23, “n” is used as a variable to denote any number of electromagneticpartitions and “m” is used to denote any level of intensity for the “n”partitions. Such a system may be cycled in the form of:

i. n at intensity m at 2306,

ii. n+1 at intensity m+1,

iii. n+2 at intensity m+2,

iv. n+i at intensity m+j at 2308.

Accordingly, any pattern of serialized cycles can be used to produce thedesired image correction wherein “i” and “j” are additional valueswithin the operation range of the imaging system.

Digital color cameras incorporate an image processing stage for thepurpose of maximizing the fidelity of color reproduction. This isaccomplished by means of a 3×3 matrix known as the Color CorrectionMatrix (CCM):

$\begin{bmatrix}R \\G \\B\end{bmatrix}_{OUT} = {\begin{bmatrix}R \\G \\B\end{bmatrix}_{IN}\begin{bmatrix}a & b & c \\d & e & f \\g & h & i\end{bmatrix}}$

The terms in the CCM are tuned using a set of reference colors (e.g.,from a Macbeth chart) to provide the best overall match to the sRGBstandard color space. The diagonal terms, a, e and i, are effectivelywhite balance gains. Typically, though, the white balance is appliedseparately, and the sums of horizontal rows are constrained to be unity,in order no net gain is applied by the CCM itself. The off-diagonalterms effectively deal with color crosstalk in the input channels.Therefore, Bayer sensors have higher off-diagonals than 3-chip camerassince the color filer arrays have a lot of response overlap betweenchannels.

There is a signal-to-noise ratio penalty for color correction which isdependent on the magnitude of the off-diagonal terms. A hypotheticalsensor with channels that perfectly matched the sRGB components wouldhave the identity matrix CCM:

$\begin{bmatrix}R \\G \\B\end{bmatrix}_{OUT} = {\begin{bmatrix}R \\G \\B\end{bmatrix}_{IN}\begin{bmatrix}1 & 0 & 0 \\0 & 1 & 0 \\0 & 0 & 1\end{bmatrix}}$

The signal to noise ratio evaluated in the green channel, for a perfectwhite photo signal of 10,000 e-per pixel (neglecting read noise) forthis case would be:

${SNR} = {\frac{\text{10,000}}{\sqrt{\text{10,000}}} = 100}$

Any departure from this degrades the SNR. Take e.g. the following CCMwhich has values that would not be unusual for a Bayer CMOS sensor:

$\begin{bmatrix}R \\G \\B\end{bmatrix}_{OUT} = {\begin{bmatrix}R \\G \\B\end{bmatrix}_{IN}\begin{bmatrix}2.6 & {- 1.4} & {- 0.2} \\{- 0.3} & 1.6 & {- 0.3} \\0 & {- 0.6} & 1.6\end{bmatrix}}$

In this case, the green SNR:

${SNR} = {\frac{\left( {{- 3000} + \text{16,000} + 3000} \right)}{\sqrt{\left( {3000 + \text{16,000} + 3000} \right)}} = 67.1}$

FIG. 24 shows the result of a full SNR simulation using D65 illuminationfor a typical Bayer sensor CCM for the case of using the identity matrixversus the tuned CCM. The SNR evaluated for the luminance component isabout 6 dB worse as a consequence of the color correction.

The system described in this disclosure uses monochromatic illuminationat a plurality of discrete wavelengths, therefore there is no colorcrosstalk per se. The crosses in FIG. 25 indicate the positions of threewavelengths which are available via laser diode sources (465, 532 & 639nm), compared the sRGB gamut which is indicated by the triangle.

The off-diagonal terms for the CCM is in this case are drasticallyreduced, compared with Bayer sensors, which provides a significant SNRadvantage.

FIG. 26 illustrates an imaging system having increased dynamic range asprovided by the pixel configuration of the pixel array of the imagesensor. As can be seen in the figure, adjacent pixels 2602 and 2604 maybe set at differing sensitivities such that each cycle includes dataproduced by pixels that are more and less sensitive with respect to eachother. Because a plurality of sensitivities can be recorded in a singlecycle of the array the dynamic range may be increased if recorded inparallel, as opposed to the time dependent serial nature of otherembodiments.

In an embodiment, an array may comprise rows of pixels that may beplaced in rows based on their sensitivities. In an embodiment, pixels ofdiffering sensitivities may alternate within a row or column withrespect to its nearest neighboring pixels to from a checkerboard patternthroughout the array based on those sensitivities. The above may beaccomplished through any pixel circuitry share arrangement or in anystand-alone pixel circuit arrangement.

Wide dynamic range can be achieved by having multiple global TX, each TXfiring only on a different set of pixels. For example, in global mode, aglobal TX1 signal is firing a set 1 of pixels, a global TX2 signal isfiring a set 2 of pixels . . . . a global TXn signal is firing a set nof pixels.

Based on FIG. 11, FIG. 27A shows a timing example for 2 different pixelsensitivities (dual pixel sensitivity) in the pixel array. In this case,global TX1 signal fires half of the pixels of the array and global TX2fires the other half of the pixels. Because global TX1 and global TX2have different “on” to “off” edge positions, integrated light isdifferent between the TX1 pixels and the TX2 pixels. FIG. 27B shows adifferent embodiment of the timing for dual pixel sensitivity. In thiscase, the light pulse is modulated twice (pulse duration and/oramplitude). TX1 pixels integrate P1 pulse and TX2 pixels integrate P1+P2pulses. Separating global TX signals can be done many ways. Thefollowing are examples:

i. Differentiating TX lines from each row; and

ii. Sending multiple TX lines per row, each addressing a different setof pixels.

In one implementation, a means of providing wide-dynamic range video isdescribed, which exploits the color pulsing system described in thisdisclosure. The basis of this is to have multiple flavors of pixels, orpixels that may be tuned differently, within the same monochrome arraythat are able to integrate the incident light for different durationswithin the same frame. An example of the pixel arrangement in the arrayof such a sensor would be a uniform checkerboard pattern throughout,with two independently variable integration times. For such a case, itis possible to provide both red and blue information within the sameframe. In fact, it is possible to do this at the same time as extendingthe dynamic range for the green frame, where it is most needed, sincethe two integration times can be adjusted on a frame by frame basis. Thebenefit is that the color motion artifacts are less of an issue if allthe data is derived from two frames versus three. There is of course asubsequent loss of spatial resolution for the red and blue data, butthat is of less consequence to the image quality compared with green,since the luminance component is dominated by green data.

An inherent property of the monochrome wide-dynamic range (WDR) array isthat the pixels that have the long integration time must integrate asuperset of the light seen by the short integration time pixels. Forregular wide-dynamic range operation in the green frames, that isdesirable. For the red and blue frames, it means that the pulsing mustbe controlled in conjunction with the exposure periods to, e.g., provideblue light from the start of the long exposure and switch to red at thepoint that the short exposure pixels are turned on (both pixel typeshave their charges transferred at the same time).

At the color fusion stage, the two flavors of pixels are separated intotwo buffers. The empty pixels are then filled in using, e.g., linearinterpolation. At this point, one buffer contains a full image of bluedata and the other red+blue. The blue buffer may be subtracted from thesecond buffer to give pure red data.

FIGS. 28A-28C illustrate the use of a white light emission that ispulsed and/or synced, or held constant, with a corresponding colorsensor. As can be seen in FIG. 28A, a white light emitter may beconfigured to emit a beam of light during the blanking period of acorresponding sensor to provide a controlled light source in acontrolled light environment. The light source may emit a beam at aconstant magnitude and vary the duration of the pulse as seen in FIG.28A, or may hold the pulse constant with varying the magnitude toachieve correctly exposed data as illustrated in FIG. 28B. Illustratedin FIG. 28C is a graphical representation of a constant light sourcethat can be modulated with varying current that is controlled by andsynced with a sensor.

In an embodiment, white light or multi-spectrum light may be emitted asa pulse, if desired, to provide data for use within the system(illustrated best in FIGS. 28A-28C). White light emissions incombination with partitions of the electromagnetic spectrum may beuseful for emphasizing and de-emphasizing certain aspects within ascene. Such an embodiment might use a pulsing pattern of:

i. Green pulse;

ii. Red pulse;

iii. Blue pulse;

iv. Green pulse;

v. Red pulse;

vi. Blue pulse;

vii. White light (multi-spectrum) pulse;

viii. (Repeat)

Any system using an image sensor cycle that is at least two times fasterthan the white light cycle is intended to fall within the scope of thedisclosure. It will be appreciated that any combination of partitions ofthe electromagnetic spectrum is contemplated herein, whether it be fromthe visible or non-visible spectrum of the full electromagneticspectrum.

FIGS. 29A and 29B illustrate a perspective view and a side view,respectively, of an implementation of a monolithic sensor 2900 having aplurality of pixel arrays for producing a three-dimensional image inaccordance with the teachings and principles of the disclosure. Such animplementation may be desirable for three-dimensional image capture,wherein the two pixel arrays 2902 and 2904 may be offset during use. Inanother implementation, a first pixel array 2902 and a second pixelarray 2904 may be dedicated to receiving a predetermined range of wavelengths of electromagnetic radiation, wherein the first pixel array isdedicated to a different range of wave length electromagnetic radiationthan the second pixel array.

FIGS. 30A and 30B illustrate a perspective view and a side view,respectively, of an implementation of an imaging sensor 3000 built on aplurality of substrates. As illustrated, a plurality of pixel columns3004 forming the pixel array are located on the first substrate 3002 anda plurality of circuit columns 3008 are located on a second substrate3006. Also illustrated in the figure are the electrical connection andcommunication between one column of pixels to its associated orcorresponding column of circuitry. In one implementation, an imagesensor, which might otherwise be manufactured with its pixel array andsupporting circuitry on a single, monolithic substrate/chip, may havethe pixel array separated from all or a majority of the supportingcircuitry. The disclosure may use at least two substrates/chips, whichwill be stacked together using three-dimensional stacking technology.The first 3002 of the two substrates/chips may be processed using animage CMOS process. The first substrate/chip 3002 may be comprisedeither of a pixel array exclusively or a pixel array surrounded bylimited circuitry. The second or subsequent substrate/chip 3006 may beprocessed using any process and does not have to be from an image CMOSprocess. The second substrate/chip 3006 may be, but is not limited to, ahighly dense digital process to integrate a variety and number offunctions in a very limited space or area on the substrate/chip, or amixed-mode or analog process to integrate for example precise analogfunctions, or a RF process to implement wireless capability, or MEMS(Micro-Electro-Mechanical Systems) to integrate MEMS devices. The imageCMOS substrate/chip 3002 may be stacked with the second or subsequentsubstrate/chip 3006 using any three-dimensional technique. The secondsubstrate/chip 3006 may support most, or a majority, of the circuitrythat would have otherwise been implemented in the first image CMOS chip3002 (if implemented on a monolithic substrate/chip) as peripheralcircuits and therefore have increased the overall system area whilekeeping the pixel array size constant and optimized to the fullestextent possible. The electrical connection between the twosubstrates/chips may be done through interconnects 3003 and 3005, whichmay be wire bonds, bump and/or TSV (Through Silicon Via).

FIGS. 31A and 31B illustrate a perspective view and a side view,respectively, of an implementation of an imaging sensor 3100 having aplurality of pixel arrays for producing a three-dimensional image. Thethree-dimensional image sensor may be built on a plurality of substratesand may comprise the plurality of pixel arrays and other associatedcircuitry, wherein a plurality of pixel columns 3104 a forming the firstpixel array and a plurality of pixel columns 3104 b forming a secondpixel array are located on respective substrates 3102 a and 3102 b,respectively, and a plurality of circuit columns 3108 a and 3108 b arelocated on a separate substrate 3106. Also illustrated are theelectrical connections and communications between columns of pixels toassociated or corresponding column of circuitry.

It will be appreciated that the teachings and principles of thedisclosure may be used in a reusable device platform, a limited usedevice platform, a re-posable use device platform, or asingle-use/disposable device platform without departing from the scopeof the disclosure. It will be appreciated that in a re-usable deviceplatform an end-user is responsible for cleaning and sterilization ofthe device. In a limited use device platform, the device can be used forsome specified amount of times before becoming inoperable. Typical newdevice is delivered sterile with additional uses requiring the end-userto clean and sterilize before additional uses. In a re-posable usedevice platform, a third-party may reprocess the device (e.g., cleans,packages and sterilizes) a single-use device for additional uses at alower cost than a new unit. In a single-use/disposable device platform adevice is provided sterile to the operating room and used only oncebefore being disposed of.

An embodiment of an emitter may employ the use of a mechanical shutterand filters to create pulsed color light. As illustrated in FIG. 32, analternate method to produce pulsed color light, using a white lightsource and a mechanical color filter and shutter system 3200. The wheelcould contain a pattern of translucent color filter windows and opaquesections for shuttering. The opaque sections would not allow lightthrough and would create a period of darkness in which the sensorread-out could occur. The white light source could be based on anytechnology: laser, LED, xenon, halogen, metal halide, or other. Thewhite light can be projected through a series of color filters 3207,3209 and 3211 of the desired pattern of colored light pulses. Oneembodiment pattern could be Red filter 3207, Green filter 3209, Bluefilter 3211, Green filter 3209. The filters and shutter system 3200could be arranged on a wheel that spins at the required frequency to bein sync with the sensor such that knowledge of the arch length and rateof rotation of the mechanical color filters 3207, 3209 and 3211 andshutters 3205 system would provide timing information for the operationof a corresponding monochromatic image sensor.

Illustrated in FIG. 33 an embodiment may comprise a pattern of onlytranslucent color filters 3307, 3309 and 3311 on a filter wheel 3300. Inthe present configuration, a different shutter may be used. The shuttercould be mechanical and could dynamically adjust the “pulse” duration byvarying is size. Alternately the shutter could be electronic andincorporated into the sensor design. A motor spinning the filter wheel3300 will need to communicate with or be controlled in conjunction withthe sensor such that knowledge of the arch length and rate of rotationof the mechanical color filters 3307, 3309 and 3311 system would providetiming information for the operation of the corresponding monochromaticimage sensor. The control system will need to know the proper colorfilter for each frame captured by the sensor so that the full-colorimage can be reconstructed properly in the ISP. A color pattern of RGBGis shown, but other colors and/or patterns could be used ifadvantageous. The relative size of the color sections is shown as equalbut could be adjusted if advantageous. The mechanical structure of thefilter is shown as a circle moving rotationally, but could berectangular with a linear movement, or a different shape with adifferent movement pattern.

As illustrated FIG. 34, an embodiment for pulsing color light mayconsist of a mechanical wheel or barrel that holds the electronics andheat sinks for Red, Green, Blue or White LEDS. The LEDs would be spacedat the distance that would be related to the rate of spin or twist ofthe barrel or wheel to allow for timing of light pulsing consistent withother embodiments in the patent. The wheel or barrel would be spun usingan electrical motor and a mechanical bracket attaching the wheel orbarrel to the electrical motor. The motor would be controlled using amicrocontroller, FPGA, DSP, or other programmable device that wouldcontain a control algorithm for proper timing as described in thepatent. There would be a mechanical opening on one side that would beoptically coupled to a fiber optic to transport the fiber to the end ofthe scopes with the methods described in the patent. This coupling couldalso have a mechanical aperture that could open and close to control theamount of light allowed down the fiber optic cable. This would be amechanical shutter device alternatively one could use the electronicshutter that is designed into a CMOS or CCD type sensor. This devicewould be difficult to control and calibrate in production but is anotherway one could get pulsed light into our system.

Illustrated in FIG. 35 is an embodiment of an emitter 3502 comprising alinear filter 3504 and shutter mechanism to provide pulsedelectromagnetic radiation. The linear filter 3504 and shutter mechanismmoves horizontally at a required frequency to filter the appropriatewavelengths of light.

Illustrating in FIG. 36 is an embodiment of an emitter 3602 comprising aprism filter 3604 and shutter mechanism to provide pulsedelectromagnetic radiation. The prism filter 3604 filters light anddelivers an output can that may include a shutter. The prism filter 3604moves at a required frequency to provide a correct color output pattern.

Additionally, the teachings and principles of the disclosure may includeany and all wavelengths of electromagnetic energy, including the visibleand non-visible spectrums, such as infrared (IR), ultraviolet (UV), andX-ray.

FIG. 37 is a schematic diagram illustrating a system 3700 for providingillumination to a light deficient environment, such as for endoscopicimaging. The system 3700 may be used in combination with any of thesystems, methods, or devices disclosed herein. The system 3700 includesa light source 3702, a controller 3704, a jumper waveguide 3706, awaveguide connector 3708, a lumen waveguide 3710, a lumen 3712, and animage sensor 3714 with accompanying optical components (such as a lens).The light source 3702 generates light that travels through the jumperwaveguide 3706 and the lumen waveguide 3710 to illuminate a scene at adistal end of the lumen 3712. The light source 3700 may be used to emitany wavelength of electromagnetic energy including visible wavelengths,infrared, ultraviolet, or other wavelengths. The lumen 3712 may beinserted into a patient's body for imaging, such as during a procedureor examination. The light is output as illustrated by dashed lines 3716.A scene illuminated by the light may be captured using the image sensor3714 and displayed for a doctor or some other medical personnel. Thecontroller 3704 may provide control signals to the light source 3702 tocontrol when illumination is provided to a scene. In one embodiment, thelight source 3702 and controller 3704 are located within a cameracontrol unit (CCU) or external console to which an endoscope isconnected. If the image sensor 3714 includes a CMOS sensor, light may beperiodically provided to the scene in a series of illumination pulsesbetween readout periods of the image sensor 3714 during what is known asa blanking period. Thus, the light may be pulsed in a controlled mannerto avoid overlapping into readout periods of the image pixels in a pixelarray of the image sensor 3714.

In one embodiment, the lumen waveguide 3710 includes a one or aplurality of optical fibers. The optical fibers may be made of alow-cost material, such as plastic to allow for disposal of the lumenwaveguide 3710 and/or other portions of an endoscope. In one embodiment,a single glass fiber having a diameter of 500 microns may be used. Thejumper waveguide 3706 may be permanently attached to the light source3702. For example, a jumper waveguide 3706 may receive light from anemitter within the light source 3702 and provide that light to the lumenwaveguide 3710 at the location of the connector 3708. In one embodiment,the jumper waveguide 106 may include one or more glass fibers. Thejumper waveguide may include any other type of waveguide for guidinglight to the lumen waveguide 3710. The connector 3708 may selectivelycouple the jumper waveguide 3706 to the lumen waveguide 3710 and allowlight within the jumper waveguide 3706 to pass to the lumen waveguide3710. In one embodiment, the lumen waveguide 3710 may be directlycoupled to a light source without any intervening jumper waveguide 3706.

FIGS. 38-40 are schematic block diagrams illustrating a light source3800 having a plurality of emitters. With regard to FIG. 38, theemitters include a first emitter 3802, a second emitter 3804, and athird emitter 3806. Additional emitters may be included, as discussedfurther below. The emitters 3802, 3804, and 3806 may include one or morelaser emitters that emit light having different wavelengths. Forexample, the first emitter 3802 may emit a wavelength that is consistentwith a blue laser, the second emitter 3804 may emit a wavelength that isconsistent with a green laser, and the third emitter 3806 may emit awavelength that is consistent with a red laser. For example, the firstemitter 3802 may include one or more blue lasers, the second emitter3804 may include one or more green lasers, and the third emitter 3806may include one or more red lasers. The emitters 3802, 3804, 3806 emitlaser beams toward a collection region 3808, which may be the locationof a waveguide, lens, or other optical component for collecting and/orproviding light to a waveguide, such as the jumper waveguide 3706 orlumen waveguide 3710 of FIG. 37.

In an implementation where a patient has been administered a reagent ordye to aid in the identification of certain tissues, structures,chemical reactions, biological processes, and so forth, the emitters3802, 3804, and 3806 may emit wavelength(s) for fluorescing the reagentsor dyes. Such wavelength(s) may be determined based on the reagents ordyes administered to the patient. In such an embodiment, the emittersmay need to be highly precise for emitting desired wavelength(s) tofluoresce or activate certain reagents or dyes.

In the embodiment of FIG. 38, the emitters 3802, 3804, 3806 each deliverlaser light to the collection region 3808 at different angles. Thevariation in angle can lead to variations where electromagnetic energyis located in an output waveguide. For example, if the light passesimmediately into a fiber bundle (glass or plastic) at the collectionregion 3808, the varying angles may cause different amounts of light toenter different fibers. For example, the angle may result in intensityvariations across the collection region 3808. Furthermore, light fromthe different emitters may not be homogenously mixed so some fibers mayreceive different amounts of light of different colors. Variation in thecolor or intensity of light in different fibers can lead to non-optimalillumination of a scene. For example, variations in delivered light orlight intensities may result at the scene and captured images.

In one embodiment, an intervening optical element may be placed betweena fiber bundle and the emitters 3802, 3804, 3806 to mix the differentcolors (wavelengths) of light before entry into the fibers or otherwaveguide. Example intervening optical elements include a diffuser,mixing rod, one or more lenses, or other optical components that mix thelight so that a given fiber receive a same amount of each color(wavelength). For example, each fiber in the fiber bundle may have asame color. This mixing may lead to the same color in each fiber butmay, in some embodiments, still result in different total brightnessdelivered to different fibers. In one embodiment, the interveningoptical element may also spread out or even out the light over thecollection region so that each fiber carries the same total amount oflight (e.g., the light may be spread out in a top hat profile). Adiffuser or mixing rod may lead to loss of light.

Although the collection region 3808 is represented as a physicalcomponent in FIG. 38, the collection region 3808 may simply be a regionwhere light from the emitters 3802, 3804, and 3806 is delivered. In somecases, the collection region 3808 may include an optical component suchas a diffuser, mixing rod, lens, or any other intervening opticalcomponent between the emitters 3802, 3804, 3806 and an output waveguide.

FIG. 39 illustrates an embodiment of a light source 3800 with emitters3802, 3804, 3806 that provide light to the collection region 3808 at thesame or substantially same angle. The light is provided at an anglesubstantially perpendicular to the collection region 3808. The lightsource 3800 includes a plurality of dichroic mirrors including a firstdichroic mirror 3902, a second dichroic mirror 3904, and a thirddichroic mirror 3906. The dichroic mirrors 3902, 3904, 3906 includemirrors that reflect a first wavelength of light but transmit (or aretransparent to) a second wavelength of light. For example, the thirddichroic mirror 3906 may reflect blue laser light provided by the thirdemitter, while being transparent to the red and green light provided bythe first emitter 3802 and the second emitter 3804, respectively. Thesecond dichroic mirror 3904 may be transparent to red light from thefirst emitter 3802, but reflective to green light from the secondemitter 3804. If other colors or wavelengths are included dichroicmirrors may be selected to reflect light corresponding to at least oneemitter and be transparent to other emitters. For example, the thirddichroic mirror 3906 reflect the light form the third emitter 3806 butis to emitters “behind” it, such as the first emitter 3802 and thesecond emitter 3804. In embodiments where tens or hundreds of emittersare present, each dichroic mirror may be reflective to a correspondingemitter and emitters in front of it while being transparent to emittersbehind it. This may allow for tens or hundreds of emitters to emitelectromagnetic energy to the collection region 3808 at a substantiallysame angle.

Because the dichroic mirrors allow other wavelengths to transmit or passthrough, each of the wavelengths may arrive at the collection region3808 from a same angle and/or with the same center or focal point.Providing light from the same angle and/or same focal/center point cansignificantly improve reception and color mixing at the collectionregion 3808. For example, a specific fiber may receive the differentcolors in the same proportions they were transmitted/reflected by theemitters 3802, 3804, 3806 and mirrors 3902, 3904, 3906. Light mixing maybe significantly improved at the collection region compared to theembodiment of FIG. 38. In one embodiment, any optical componentsdiscussed herein may be used at the collection region 3808 to collectlight prior to providing it to a fiber or fiber bundle.

FIG. 40 illustrates an embodiment of a light source 3800 with emitters3802, 3804, 3806 that also provide light to the collection region 3808at the same or substantially same angle. However, the light incident onthe collection region 3808 is offset from being perpendicular. Angle4002 indicates the angle offset from perpendicular. In one embodiment,the laser emitters 3802, 3804, 3806 may have cross sectional intensityprofiles that are Gaussian. As discussed previously, improveddistribution of light energy between fibers may be accomplished bycreating a more flat or top-hat shaped intensity profile. In oneembodiment, as the angle 4002 is increased, the intensity across thecollection region 3808 approaches a top hat profile. For example, atop-hat profile may be approximated even with a non-flat output beam byincreasing the angle 4002 until the profile is sufficiently flat.

The top hat profile may also be accomplished using one or more lenses,diffusers, mixing rods, or any other intervening optical componentbetween the emitters 3802, 3804, 3806 and an output waveguide, fiber, orfiber optic bundle.

FIG. 41 is a schematic diagram illustrating a single optical fiber 4102outputting via a diffuser 4104 at an output. In one embodiment, theoptical fiber 4102 may have a diameter of 500 microns and have anumerical aperture of 0.65 and emit a light cone 4106 of about 70 or 80degrees without a diffuser 4104. With the diffuser 4104, the light cone4106 may have an angle of about 110 or 120 degrees. The light cone 4106may be a majority of where all light goes and is evenly distributed. Thediffuser 4104 may allow for more even distribution of electromagneticenergy of a scene observed by an image sensor.

In one embodiment, the lumen waveguide 4102 may include a single plasticor glass optical fiber of about 500 microns. The plastic fiber may below cost, but the width may allow the fiber to carry a sufficient amountof light to a scene, with coupling, diffuser, or other losses. Forexample, smaller fibers may not be able to carry as much light or poweras a larger fiber. The lumen waveguide 3710 may include a single or aplurality of optical fibers. The lumen waveguide 3702 may receive lightdirectly from the light source or via a jumper waveguide (e.g., see thejumper waveguide 3706 of FIG. 37). A diffuser may be used to broaden thelight output 3706 for a desired field of view of the image sensor 3714or other optical components.

Although three emitters are shown in FIGS. 38-40, emitters numberingfrom one into the hundreds or more may be used in some embodiments. Theemitters may have different wavelengths or spectrums of light that theyemit, and which may be used to contiguously cover a desired portion ofthe electromagnetic spectrum (e.g., the visible spectrum as well asinfrared and ultraviolet spectrums).

In one embodiment, a light source with a plurality of emitters may beused for multispectral or hyperspectral imaging in a light deficientenvironment. For example, different chemicals, materials, or tissue mayhave different responses to different colors or wavelengths ofelectromagnetic energy. Some tissues have their own spectral signature(how they respond or vary in reflecting wavelengths of electromagneticradiation). In one embodiment, a specific type of tissues may bedetected based on how it responds to a specific wavelength or a specificcombination of wavelengths. For example, blood vessel tissues may absorband reflect different wavelengths or spectrums of electromagnetic energyin a unique way to distinguish it from muscle, fat, bone, nerve,ureter,or other tissues or materials in the body. Furthermore, specifictypes of muscle or other types of tissue may be distinguished based ontheir spectral response. Disease states of tissue may also be determinedbased on spectral information. See U.S. Pat. No. 8,289,503. See alsoU.S. Pat. No. 8,158,957.

In one embodiment, fluorescent image data, and/or multispectral orhyperspectral image data may be obtained using one or more filters tofilter out all light or electromagnetic energy, except that in thedesired wavelength or spectrum. FIG. 42 is a block diagram illustratinga filter 4202 for filtering out unwanted wavelengths before light 4208(or other electromagnetic radiation) encounters an imaging sensor 4204or other imaging medium (e.g., film). In one embodiment, white light4208 passes through the filter 4202 and filtered light 4210 passesthrough a lens 4206 to be focused onto the imaging sensor 4204 for imagecapture and readout. The filter may be located anywhere in the system ormay be an attribute of the lens 4206 or image sensor 4204.

In a light deficient environment, the light 4208 may include white lightemitted by an emitter in the light deficient environment. The filter4202 may be selected for the desired examination. For example, if it isdesired to detect or highlight a specific tissue, the filter 4202 may beselected to allow wavelengths corresponding to the spectral response ofthe specific tissue or the fluorescence emission of a specific reagentto pass through. The image sensor 4204, which may include amonochromatic image sensor, may generate an image. Pixels of thecaptured image that exceed a threshold or fall below a threshold maythen be characterized as corresponding to the specific tissue. This datamay then be used to generate an image that indicates the location of thespecific tissue.

In another embodiment, a fluorescing dye or reagent may be used forimaging specific tissue types, pathways, or the like in a body. Forexample, a fluorescing dye may be administered to a patient and then animage of the dye may be captured. In one embodiment, fluorescing of thedye may be triggered using a specific wavelength of electromagneticenergy. For example, the dye may only fluoresce when the electromagneticenergy is present.

However, both filters and fluorescing dyes significantly constrainexamination. For example, if a filter is used, the desired spectralresponse that can be detected, and thus the material or tissue that canbe detected, is limited by the available filters. Furthermore, thefilters may need to be swapped or replaced. With regard to dyes, the dyemust be administered before imaging and there may be conflicts betweenadministering different dyes for different purposes during the sameexamination. Thus, examinations using filters and dyes can take a longtime and may require many different examinations to get the desiredinformation.

In one embodiment, multispectral or hyper spectral imaging in a lightdeficient environment may be achieved using a monochrome image sensorand emitters that emit a plurality of different wavelengths or spectrumsof electromagnetic energy. In one embodiment, a light source or otherelectromagnetic source (such as a light source 3800 in any of FIGS.38-40) may include a plurality of emitters to cover desired spectrums.

FIG. 43 illustrates a portion of the electromagnetic spectrum 4300divided into twenty different sub-spectrums. The number of sub-spectrumsis illustrative only. In at least one embodiment, the spectrum 4300 maybe divided into hundreds of sub-spectrums, each with a small waveband.The spectrum may extend from the infrared spectrum 4302, through thevisible spectrum 4304, and into the ultraviolet spectrum 4306. Thesub-spectrums each have a waveband 4308 that covers a portion of thespectrum 4300. Each waveband may be defined by an upper wavelength and alower wavelength.

In one embodiment, at least one emitter (such as a laser emitter) may beincluded in a light source (such as the light sources 3702, 3800 inFIGS. 37-40) for each sub-spectrum to provide complete and contiguouscoverage of the whole spectrum 4300. For example, a light source forproviding coverage of the illustrated sub-spectrums may include at least20 different emitters, at least one for each sub-spectrum. In oneembodiment, each emitter may cover a spectrum covering 40 nanometers.For example, one emitter may emit light within a waveband from 500 nm to540 nm while another emitter may emit light within a waveband from 540nm to 580 nm. In another embodiment, emitters may cover other sizes ofwavebands, depending on the types of emitters available or the imagingneeds. For example, a plurality of emitters may include a first emitterthat covers a waveband from 500 to 540 nm, a second emitter that coversa waveband from 540 nm to 640 nm, and a third emitter that covers awaveband from 640 nm to 650 nm. Each emitter may cover a different sliceof the electromagnetic spectrum ranging from far infrared, mid infrared,near infrared, visible light, near ultraviolet and/or extremeultraviolet. In some cases, a plurality of emitters of the same type orwavelength may be included to provide sufficient output power forimaging. The number of emitters needed for a specific waveband maydepend on the sensitivity of a monochrome sensor to the waveband and/orthe power output capability of emitters in that waveband.

The waveband widths and coverage provided by the emitters may beselected to provide any desired combination of spectrums. For example,contiguous coverage of a spectrum using very small waveband widths(e.g., 10 nm or less) may allow for highly selective hyperspectralimaging. Because the wavelengths come from emitters which can beselectively activated, extreme flexibility in determining spectralresponses of a material during an examination can be achieved. Thus,much more information about spectral response may be achieved in lesstime and within a single examination which would have required multipleexaminations, delays because of the administration of dyes or stains, orthe like. In one embodiment, a system may capture hyperspectral imagedata and process that data to identify what type of tissue exists ateach pixel.

FIG. 44 is a schematic diagram illustrating a timing diagram 4400 foremission and readout for generating a multispectral or hyperspectralimage, according to one embodiment. The solid line represents readout(peaks 4402) and blanking periods (valleys) for capturing a series offrames 4404-4414. The series of frames 4404-4414 may include a repeatingseries of frames which may be used for generating hyperspectral data fora video feed. The series of frames include a first frame 404, a secondframe 4406, a third frame 4408, a fourth frame 4410, a fifth frame 4412,and an Nth frame 4426.

In one embodiment, each frame is generated based on at least one pulseof electromagnetic energy. The pulse of electromagnetic energy isreflected and detected by an image sensor and then read out in asubsequent readout (4402). Thus, each blanking period and readoutresults in an image frame for a specific spectrum of electromagneticenergy. For example, the first frame 404 may be generated based on aspectrum of a first one or more pulses 4416, a second frame 4406 may begenerated based on a spectrum of a second one or more pulses 4418, athird frame 4408 may be generated based on a spectrum of a third one ormore pulses 4420, a fourth frame 4410 may be generated based on aspectrum of a fourth one or more pulses 4422, a fifth frame 4412 may begenerated based on a spectrum of a fifth one or more pulses 4424, and anNth frame 4426 may be generated based on a spectrum of an Nth one ormore pulses 4426.

The pulses 4416-4426 may include energy from a single emitter or from acombination of two or more emitters. For example, the spectrum includedin a single readout period or within the plurality of frames 4404-4414may be selected for a desired examination or detection of a specifictissue or condition. According to one embodiment, one or more pulses mayinclude visible spectrum light for generating a color or black and whiteimage while one or more additional pulses are used to obtain spectralresponse to classify a type of tissue. For example, pulse 4416 mayinclude red light, pulse 4418 may include blue light, and pulse 4420 mayinclude green light while the remaining pulses 4422-4426 may includewavelengths and spectrums for detecting a specific tissue type. As afurther example, pulses for a single readout period may include aspectrum generated from multiple different emitters (e.g., differentslices of the electromagnetic spectrum) that can be used to detect aspecific tissue type. For example, if the combination of wavelengthsresults in a pixel having a value exceeding or falling below athreshold, that pixel may be classified as corresponding to a specifictype of tissue. Each frame may be used to further narrow the type oftissue that is present at that pixel (e.g., and each pixel in the image)to provide a very specific classification of the tissue and/or a stateof the tissue (diseased/healthy) based on the spectral response.

The plurality of frames 4404-4414 is shown having varying lengths inreadout periods and pulses having different lengths or intensities. Theblanking period, pulse length or intensity, or the like may be selectedbased on the sensitivity of a monochromatic sensor to the specificwavelength, the power output capability of the emitter(s), and/or thecarrying capacity of the waveguide.

A hyperspectral image or hyperspectral image data obtained in a mannerillustrated in FIG. 44 may result in a plurality of frames, each basedon a different spectrum or combination of spectrums. In some cases, tensor hundreds of different frames may be obtained. In other cases, such asfor video streams, the number of frames may be limited to provide aviewable frame rate. Because combinations of different spectrums may beprovided in a single readout period, useful and dynamic spectralinformation may still be obtained even in a video stream.

In one embodiment, a video or other image may include a black and whiteor color image overlaid with information derived from the spectralresponse for each pixel. For example, pixels that correspond to aspecific tissue or state may be shown in a bright green or other colorto assist a doctor or other medical expert during an examination.

In one embodiment, dual image sensors may be used to obtainthree-dimensional images or video feeds. A three-dimensional examinationmay allow for improved understanding of a three-dimensional structure ofthe examined region as well as a mapping of the different tissue ormaterial types within the region.

In one embodiment, multispectral or hyperspectral imaging may be used tolook through materials or substances. For example, infrared wavelengthsmay pass through some tissues, such as muscle or fat, while reflectingoff blood vessels. In one embodiment, infrared waves may penetrate 5, 8or 10 mm or more into a tissue. Obtaining a series of frames thatincludes at least one infrared frame may allow an examination to provideinformation about the location of blood vessels below the surface. Thiscan be extremely helpful for surgical procedures where it may bedesirable to perform incisions that avoid blood vessels. In oneembodiment, a color or greysc ale image may be overlaid with a greencolor that indicates the location of blood vessels below the surface.Similarly, a known spectral response of blood may be used to lookthrough the blood and see the tissues or structures of interest in anexamination.

Assembly of the subframes into a single frame for display on a monitoror other display device may take place after capturing the series offrames 4404-4414. A color or greyscale image may be generated from oneor more of the frames and overlay information for pixels may bedetermined based on all or the remaining frames. The color or greyscaleimage mat be combined with the overlay information to generate a singleframe. The single frame may be displayed as a single image or as animage in a video stream.

In one embodiment, the hyperspectral data obtained as illustrated inFIG. 44 may be provided for analysis by a third-party algorithm toclassify a tissue or material captured in the image. In one embodiment,the third-party algorithm may be used to select the spectrums orwavebands to be used during imaging so that a desired spectral responseanalysis can be performed. In an embodiment, the spectral responseanalysis may be performed in real-time during a medical imagingprocedure or other medical procedure. The spectral data may be overlaidon an RGB or black and white image such that a user may readilydifferentiate certain types of tissues, organs, chemical processes,diseases, and so forth. In an embodiment, the spectral data may beprovided to a computer-operated system, such as a robotics system, forautomation of medical imaging or medical procedures.

FIG. 45 is a schematic diagram of an imaging system 4500 having a singlecut filter. The system 4500 includes an endoscope 4506 or other suitableimaging device having a light source 4508 for use in a light deficientenvironment. The endoscope 4506 includes an image sensor 4504 and afilter 4502 for filtering out unwanted wavelengths of light or otherelectromagnetic radiation before reaching the image sensor 4504. Thelight source 4508 transmits light that may illuminate the surface 4512in a light deficient environment such as a body cavity. The light 4510is reflected off the surface 4512 and passes through the filter 4502before hitting the image sensor 4504.

The filter 4502 may be used in an implementation where a fluorescentreagent or dye has been administered. In such an embodiment, the filter4502 is configured to filter out all light other than one or moredesired wavelengths or spectral bands of light or other electromagneticradiation. In one embodiment, the filter 4502 is configured to filterout an excitation wavelength of electromagnetic radiation that causes areagent or dye to fluoresce such that only the expected relaxationwavelength of the fluoresced reagent or dye is permitted to pass throughthe filter 4502 and reach the image sensor 4504. In an embodiment, thefilter 4502 filters out at least a fluorescent reagent excitationwavelength between 770 nm and 790 nm. In an embodiment, the filter 4502filters out at least a fluorescent reagent excitation wavelength between795 nm and 815 nm. In an embodiment, the filter 4502 filters out atleast a fluorescent reagent excitation wavelength between 770 nm and 790nm and between 795 nm and 815 nm. In these embodiments, the filter 4502filters out the excitation wavelength of the reagent and permits onlythe relaxation wavelength of the fluoresced reagent to be read by theimage sensor 4504. The image sensor 4504 may be a wavelength-agnosticimage sensor and the filter 4502 may be configured to permit the imagesensor 4504 to only receive the relaxation wavelength of the fluorescedreagent and not receive the emitted excitation wavelength for thereagent. The data determined by the image sensor 4504 may then indicatea presence of a critical body structure, tissue, biological process, orchemical process as determined by a location of the reagent or dye.

The filter 4502 may further be used in an implementation where afluorescent reagent or dye has not been administered. The filter 4502may be selected to permit wavelengths corresponding to a desiredspectral response to pass through and be read by the image sensor 4504.The image sensor 4504 may be a monochromatic image sensor such thatpixels of the captured image that exceed a threshold or fall below athreshold may be characterized as corresponding to a certain spectralresponse or fluorescence emission. The spectral response or fluorescenceemission, as determined by the pixels captured by the image sensor 4504,may indicate the presence of a certain body tissue or structure, acertain condition, a certain chemical process, and so forth.

In one embodiment, the light source 4508 transmits white light thatcontacts the surface 4512 and is reflected back where it is filtered bythe filter 4502 before it hits the image sensor 4504. In one embodiment,the light source 4508 transmits white light that passes through thefilter 4502 such that filtered light of only one or more desiredwavelengths emerges from the filter 4502 to be reflected off the surface4512 and read by the image sensor 4504. For example, in an embodiment,the filter 4502 permits only light having a wavelength of 795 nm to passthrough the filter 4502 and contact the image sensor 4504. Further in anembodiment, the filter 4502 permits only certain wavelengths of light tobe reflected back to the image sensor 4504 of the endoscope 4506 orother imaging device. The filter 4502 may be located anywhere in thesystem 4500 or may be an attribute of a lens or the image sensor 4504.The filter 4502 may be located in front of and/or behind the imagesensor 4504. In an embodiment, light emitted by the light source 4508 isfiltered before it reaches the surface 4512 and the reflected light isfiltered by an additional filter before it is ready by the image sensor4504.

The light source 4508 may be an emitter that may be configured to emitwhite light or electromagnetic radiation of one or more specificwavelengths. The light source 4508 may include a plurality of lasersconfigured to emit or pulse light of specified wavelengths. In anembodiment, the light source 4508 emits white light and the filter 4502is selected to filter all unwanted light other than one or more desiredwavelengths of light or other electromagnetic radiation. The filter 4502may be selected for a specific examination or purpose, for example tohighlight a type of body tissue or structure, or to highlight a certaincondition or chemical process.

FIG. 46 is a schematic diagram of an imaging system 4600 having multiplecut filters. The system 4600 includes an endoscope 4606 or othersuitable imaging device having a light source 4608 for use in a lightdeficient environment. The endoscope 4606 includes an image sensor 4604and two filters 4602 a, 4602 b. It should be appreciated that inalternative embodiments, the system 4600 may include any number offilters, and the number of filters and the type of filters may beselected for a certain purpose e.g., for gathering imaging informationof a particular body tissue, body condition, chemical process, and soforth. The filters 4602 a, 4602 b are configured for filtering outunwanted wavelengths of light or other electromagnetic radiation. Thefilters 4602 a, 4602 b may be configured to filter out unwantedwavelengths from white light or other electromagnetic radiation that maybe emitted by the light source 4608. The filtered light may hit thesurface 4612 (e.g. body tissue) and be reflected back on to the imagesensor 4604.

Further to the disclosure with respect to FIG. 45, the filters 4602 a,4602 b may be used in an implementation where a fluorescent reagent ordye has been administered. The filters 4602 a, 4602 b may be configuredfor blocking an emitted excitation wavelength for the reagent or dye andpermitting the image sensor 4604 to only read the relaxation wavelengthof the reagent or dye. Further, the filters 4602 a, 4602 b may be usedin an implementation where a fluorescent reagent or dye has not beenadministered. In such an implementation, the filters 4602 a, 4602 b maybe selected to permit wavelengths corresponding to a desired spectralresponse to pass through and be read by the image sensor 4604.

The multiple filters 4602 a, 4602 b may each be configured for filteringout a different range of wavelengths of the electromagnetic spectrum.For example, one filter may be configured for filtering out wavelengthslonger than a desired wavelength range and the additional filter may beconfigured for filtering out wavelengths shorter than the desiredwavelength range. The combination of the two or more filters may resultin only a certain wavelength or band of wavelengths being read by theimage sensor 4604.

In an embodiment, the filters 4602 a, 4602 b are customized such thatelectromagnetic radiation between 513 nm and 545 nm contacts the imagesensor 4604. In an embodiment, the filters 4602 a, 4602 b are customizedsuch that electromagnetic radiation between 565 nm and 585 nm contactsthe image sensor 4604. In an embodiment, the filters 4602 a, 4602 b arecustomized such that electromagnetic radiation between 900 nm and 1000nm contacts the image sensor 4604. In an embodiment, the filters 4602 a,4602 b are customized such that electromagnetic radiation between 425 nmand 475 nm contacts the image sensor 4604. In an embodiment, the filters4602 a, 4602 b are customized such that electromagnetic radiationbetween 520 nm and 545 nm contacts the image sensor 4604. In anembodiment, the filters 4602 a, 4602 b are customized such thatelectromagnetic radiation between 625 nm and 645 nm contacts the imagesensor 4604. In an embodiment, the filters 4602 a, 4602 b are customizedsuch that electromagnetic radiation between 760 nm and 795 nm contactsthe image sensor 4604. In an embodiment, the filters 4602 a, 4602 b arecustomized such that electromagnetic radiation between 795 nm and 815 nmcontacts the image sensor 4604. In an embodiment, the filters 4602 a,4602 b are customized such that electromagnetic radiation between 370 nmand 420 nm contacts the image sensor 4604. In an embodiment, the filters4602 a, 4602 b are customized such that electromagnetic radiationbetween 600 nm and 670 nm contacts the image sensor 4604. In anembodiment, the filters 4602 a, 4602 b are configured for permittingonly a certain fluorescence relaxation emission to pass through thefilters 4602 a, 4602 b and contact the image sensor 4604.

In an embodiment, the system 4600 includes multiple image sensors 4604and may particularly include two image sensors for use in generating athree-dimensional image. The image sensor(s) 4604 may becolor/wavelength agnostic and configured for reading any wavelength ofelectromagnetic radiation that is reflected off the surface 4612. In anembodiment, the image sensors 4604 are each color dependent orwavelength dependent and configured for reading electromagneticradiation of a particular wavelength that is reflected off the surface4612 and back to the image sensors 4604. Alternatively, the image sensor4604 may include a single image sensor with a plurality of differentpixel sensors configured for reading different wavelengths or colors oflight, such as a Bayer filter color filter array. Alternatively, theimage sensor 4604 may include one or more color agnostic image sensorsthat may be configured for reading different wavelengths ofelectromagnetic radiation according to a pulsing schedule such as thoseillustrated in FIGS. 5-7E and 15-16, for example.

FIG. 47 is a schematic diagram illustrating a system 4700 for mapping asurface and/or tracking an object in a light deficient environment. Inan embodiment, an endoscope 4702 in a light deficient environment pulsesa grid array 4706 (may be referred to as a laser map pattern) on asurface 4704. The grid array 4706 may include vertical hashing 4708 andhorizontal hashing 4710 in one embodiment as illustrated in FIG. 47. TheIt should be appreciated the grid array 4706 may include any suitablearray for mapping a surface 4704, including, for example, a raster gridof discrete points, an occupancy grid map, a dot array, and so forth.Additionally, the endoscope 4702 may pulse multiple grid arrays 4706 andmay, for example, pulse one or more individual grid arrays on each of aplurality of objects or structures within the light deficientenvironment.

In an embodiment, the system 4700 pulses a grid array 4706 that may beused for determining a three-dimensional surface and/or tracking alocation of an object such as a tool or another device in a lightdeficient environment. In an embodiment, the system 4700 may providedata to a third party system or computer algorithm for determiningsurface dimensions and configurations by way of light detection andranging (LIDAR) mapping. The system 4700 may pulse any suitablewavelength of light or electromagnetic radiation in the grid array 4706,including, for example, ultraviolet light, visible, light, and/orinfrared or near infrared light. The surface 4704 and/or objects withinthe environment may be mapped and tracked at very high resolution andwith very high accuracy and precision.

In an embodiment, the system 4700 includes an imaging device having atube, one or more image sensors, and a lens assembly having an opticalelement corresponding to the one or more image sensors. The system 4700may include a light engine having an illumination source generating oneor more pulses of electromagnetic radiation and a lumen transmitting theone or more pulses of electromagnetic radiation to a distal tip of anendoscope within a light deficient environment such as a body cavity. Inan embodiment, at least a portion of the one or more pulses ofelectromagnetic radiation includes a laser map pattern that is emittedonto a surface within the light deficient environment, such as a surfaceof body tissue and/or a surface of tools or other devices within thebody cavity. The endoscope 4702 may include a two-dimensional,three-dimensional, or n-dimensional camera for mapping and/or trackingthe surface, dimensions, and configurations within the light deficientenvironment.

In an embodiment, the system 4700 includes a processor for determining adistance of an endoscope or tool from an object such as the surface4704. The processor may further determine an angle between the endoscopeor tool and the object. The processor may further determine surface areainformation about the object, including for example, the size ofsurgical tools, the size of structures, the size of anatomicalstructures, location information, and other positional data and metrics.The system 4700 may include one or more image sensors that provide imagedata that is output to a control system for determining a distance of anendoscope or tool to an object such as the surface 4704. The imagesensors may output information to a control system for determining anangle between the endoscope or tool to the object. Additionally, theimage sensors may output information to a control system for determiningsurface area information about the object, the size of surgical tools,size of structures, size of anatomical structures, location information,and other positional data and metrics.

In an embodiment, the grid array 4706 is pulsed by an illuminationsource of the endoscope 4702 at a sufficient speed such that the gridarray 4706 is not visible to a user. In various implementations, it maybe distracting to a user to see the grid array 4706 during an endoscopicimaging procedure and/or endoscopic surgical procedure. The grid array4706 may be pulsed for sufficiently brief periods such that the gridarray 4706 cannot be detected by a human eye. In an alternativeembodiment, the endoscope 4702 pulses the grid array 4706 at asufficient recurring frequency such that the grid array 4706 may beviewed by a user. In such an embodiment, the grid array 4706 may beoverlaid on an image of the surface 4704 on a display. The grid array4706 may be overlaid on a black-and-white or RGB image of the surface4704 such that the grid array 4706 may be visible by a user during useof the system 4700. A user of the system 4700 may indicate whether thegrid array 4706 should be overlaid on an image of the surface 4704and/or whether the grid array 4706 should be visible to the user. Thesystem 4700 may include a display that provides real-time measurementsof a distance from the endoscope 4702 to the surface 4704 or anotherobject within the light deficient environment. The display may furtherprovide real-time surface area information about the surface 4704 and/orany objects, structures, or tools within the light deficientenvironment. The accuracy of the measurements may be accurate to lessthan one millimeter.

The endoscope 4702 may pulse electromagnetic radiation according to apulsing schedule such as those illustrated in FIGS. 5-7E and 15-16, forexample, that may further include pulsing of the grid array 4706 alongwith pulsing Red, Green, and Blue light for generating an RGB image andfurther generating a grid array 4706 that may be overlaid on the RGBimage and/or used for mapping and tracking the surface 4704 and objectswithin the light deficient environment.

In an embodiment, the endoscope 4702 includes one or more color agnosticimage sensors. In an embodiment, the endoscope 4702 includes two coloragnostic image sensors for generating a three-dimensional image or mapof the light deficient environment. The image sensors may generate anRGB image of the light deficient environment according to a pulsingschedule as disclosed herein. Additionally, the image sensors maydetermine data for mapping the light deficient environment and trackingone or more objects within the light deficient environment based on datadetermined when the grid array 4706 is pulsed. Additionally, the imagesensors may determine spectral or hyperspectral data along withfluorescence imaging data according to a pulsing schedule that may bemodified by a user to suit the particular needs of an imaging procedure.In an embodiment, a pulsing schedule includes Red, Green, and Bluepulses along with pulsing of a grid array 4706 and/or pulsing forgenerating hyperspectral image data and/or fluorescence image data. Invarious implementations, the pulsing schedule may include any suitablecombination of pulses of electromagnetic radiation according to theneeds of a user. The recurring frequency of the different wavelengths ofelectromagnetic radiation may be determined based on, for example, theenergy of a certain pulse, the needs of the user, whether certain data(for example, hyperspectral data and/or fluorescence imaging data) needsto be continuously updated or may be updated less frequently, and soforth.

The pulsing schedule may be modified in any suitable manner, and certainpulses of electromagnetic radiation may be repeated at any suitablefrequency, according to the needs of a user or computer-implementedprogram for a certain imaging procedure. For example, in an embodimentwhere surface tracking data generated based on the grid array 4706 isprovided to a computer-implemented program for use in, for example, arobotic surgical procedure, the grid array 4706 may be pulsed morefrequently than if the surface tracking data is provided to a user whois visualizing the scene during the imaging procedure. In such anembodiment where the surface tracking data is used for a roboticsurgical procedure, the surface tracking data may need to be updatedmore frequently or may need to be exceedingly accurate such that thecomputer-implemented program may execute the robotic surgical procedurewith precision and accuracy.

In an embodiment, the system 4700 is configured to generate an occupancygrid map comprising an array of cells divided into grids. The system4700 is configured to store height values for each of the respectivegrid cells to determine a surface mapping of a three-dimensionalenvironment in a light deficient environment.

FIG. 48 is a schematic flow chart diagram for a method 4800 forhyperspectral imaging in a light deficient environment. The method 4800may be performed by an imaging system, such as an endoscopic imagingsystem illustrated in FIG. 37.

The method 4800 includes emitting at 4802 a plurality of narrow bandpulses during readout periods of a monochromatic image sensor. Thepulses may be emitted at 4802 using a light source that includes aplurality of emitters that emit electromagnetic energy within the narrowfrequency bands. For example, the light source may include at least oneemitter for a plurality of frequency bands covering a desired spectrum.A monochromatic image sensor reads out at 4804 pixel data from themonochromatic image sensor following the readout periods to generate aplurality of frames. Each frame may include a different spectralcontent. These frames may include a plurality of repeating frames thatmay be used for generating a digital video stream. Each frame may bebased energy emitted by one or more emitters of the light source. In oneembodiment, a frame may be based on a combination of light emitted bylight sources to generate a combination of frequencies to match afrequency response of a desired tissue or substance. A controller, CCU,or other system determines at 4806 a spectral response of a tissue forone or more pixels based on the plurality of frames. For example, thepixel values and knowledge about the frequencies of light emitted foreach frame may be used to determine a frequency response for a specificpixel, based on the values for the pixel in the plurality of frames. Thesystem may generate at 4808 a combined image based on the plurality offrames, the combined image comprising an overlay indicating the spectralresponse for the one or more pixels. For example, the combined image maybe a greyscale or color image where pixels corresponding to a specifictissue or classification are shown in bright green.

FIG. 49 is a schematic flow chart diagram for a method 4900 forfluorescence imaging in a light deficient environment. The method 4900may be performed by an imaging system, such as an endoscopic imagingsystem illustrated in FIG. 37.

The method 4900 includes emitting at 4902 a plurality of narrow bandpulses during readout periods of a monochromatic image sensor. Thepulses may be emitted at 4902 using a light source that includes aplurality of emitters that emit electromagnetic energy within the narrowfrequency bands. For example, the light source may include at least oneemitter for a plurality of frequency bands covering a desired spectrum.A monochromatic image sensor reads out at 4904 pixel data from themonochromatic image sensor following the readout periods to generate aplurality of frames. Each frame may include a different spectralcontent. These frames may include a plurality of repeating frames thatmay be used for generating a digital video stream. Each frame may bebased energy emitted by one or more emitters of the light source. In oneembodiment, a frame may be based on a combination of light emitted bylight sources to generate a combination of frequencies to match afrequency response of a desired tissue or substance. A controller, CCU,or other system determines at 4906 a fluorescence relaxation emission ofa reagent for one or more pixels based on the plurality of frames. Forexample, the pixel values and knowledge about the frequencies of lightemitted for each frame may be used to determine a frequency response fora specific pixel, based on the values for the pixel in the plurality offrames. The system may generate at 4908 a combined image based on theplurality of frames, the combined image comprising an overlay indicatingthe fluorescence relaxation emission for the one or more pixels. Forexample, the combined image may be a greysc ale or color image wherepixels corresponding to a specific tissue or classification are shown inbright green.

EXAMPLES

The following examples pertain to further embodiments:

Example 1 is an endoscopic system for use in a light deficientenvironment. The system includes an imaging device. The imaging deviceincludes a tube, one or more image sensors, and a lens assemblycomprising at least one optical element corresponding to the imagesensor. The system includes a display for a user to visualize a sceneand an image signal processing controller. The system includes a lightengine. The light engine includes an illumination source generating oneor more pulses of electromagnetic radiation. The light engine furtherincludes a lumen transmitting one or more pulses of electromagneticradiation to a distal tip of an endoscope, wherein at least a portion ofthe one or more pulses of electromagnetic radiation includes anexcitation wavelength of electromagnetic radiation between 770 nm and790 nm that causes a reagent to fluoresce at a wavelength that isdifferent from the excitation wavelength of the portion of the one ormore pulses of electromagnetic radiation.

Example 2 is an endoscopic system as in Example 1, wherein the systemfurther comprises a filter that blocks the excitation wavelength ofelectromagnetic radiation between 770 nm and 790 nm.

Example 3 is an endoscopic system as in any of Examples 1-2, wherein thefilter is located on the at least one optical element of the lensassembly, such that the filter blocks the excitation wavelength andallows a wavelength of the fluorescing reagent through the filter.

Example 4 is an endoscopic system as in any of Examples 1-3, whereineach pulse of electromagnetic radiation results in an exposure framecreated by the image sensor; wherein one or more exposure frames aredisplayed to a user as a single image on the display.

Example 5 is an endoscopic system as in any of Examples 1-4, wherein thesingle image is assigned a visible color for use on the display; whereinthe visible color is 8-bit or 16-bit or n-bit.

Example 6 is an endoscopic system as in any of Examples 1-5, whereineach pulse of electromagnetic radiation results in an exposure framecreated by the image sensor; wherein one or more exposure frames aredisplayed to a user as an overlay image on the display.

Example 7 is an endoscopic system as in any of Examples 1-6, wherein theoverlay image is assigned a visible color for use on the display;wherein the visible color is 8-bit or 16-bit or n-bit.

Example 8 is an endoscopic system as in any of Examples 1-7, wherein theimage sensor detects a wavelength of the fluorescing reagent to providean image of one or more critical structures in a human body.

Example 9 is an endoscopic system as in any of Examples 1-8, wherein thecritical structures in a human body include one of a nerve, a ureter, ablood vessel, an artery, a blood flow, and a tumor.

Example 10 is an endoscopic system as in any of Examples 1-9, whereinthe one or more critical structures are cancer cells, and wherein thesystem receives fluoresced electromagnetic radiation from a moleculethat attaches a fluorophore that fluoresces when exposed toelectromagnetic radiation having a wavelength between 770 nm and 790 nmto one or more of the cancer cells.

Example 11 is an endoscopic system as in any of Examples 1-10, whereineach pulse of electromagnetic radiation results in an exposure framecreated by the image sensor; wherein one or more exposure frames aredisplayed to a user as a single image on the display.

Example 12 is an endoscopic system as in any of Examples 1-11, whereinthe single image is assigned a visible color for use on the display;wherein the visible color is 8-bit or 16-bit or n-bit.

Example 13 is an endoscopic system as in any of Examples 1-12, whereineach pulse of electromagnetic radiation results in an exposure framecreated by the image sensor; wherein one or more exposure frames aredisplayed to a user as an overlay image on the display.

Example 14 is an endoscopic system as in any of Examples 1-13, whereinthe overlay image is assigned a visible color for use on the display;wherein the visible color is 8-bit or 16-bit or n-bit.

Example 15 is an endoscopic system as in any of Examples 1-14, whereinthe illumination source generates one or more pulses of electromagneticradiation at a wavelength of 370 nm to 420 nm.

Example 16 is an endoscopic system as in any of Examples 1-15, whereineach pulse of electromagnetic radiation results in an exposure framecreated by the image sensor; wherein one or more exposure frames aredisplayed to a user as a single image on the display.

Example 17 is an endoscopic system as in any of Examples 1-16, whereinthe single image is assigned a visible color for use on the display;wherein the visible color is 8-bit or 16-bit or n-bit.

Example 18 is an endoscopic system as in any of Examples 1-17, whereineach pulse of electromagnetic radiation results in an exposure framecreated by the image sensor; wherein one or more exposure frames aredisplayed to a user as an overlay image on the display.

Example 19 is an endoscopic system as in any of Examples 1-18, whereinthe overlay image is assigned a visible color for use on the display;wherein the visible color is 8-bit or 16-bit or n-bit.

Example 20 is an endoscopic system as in any of Examples 1-19, whereinthe illumination source generates one or more pulses of electromagneticradiation at a wavelength of 600 nm to 670 nm.

Example 21 is an endoscopic system as in any of Examples 1-20, whereineach pulse of electromagnetic radiation results in an exposure framecreated by the image sensor; wherein one or more exposure frames aredisplayed to a user as a single image on the display.

Example 22 is an endoscopic system as in any of Examples 1-21, whereinthe single image is assigned a visible color for use on the display;wherein the visible color is 8-bit or 16-bit or n-bit.

Example 23 is an endoscopic system as in any of Examples 1-22, whereineach pulse of electromagnetic radiation results in an exposure framecreated by the image sensor; wherein one or more exposure frames aredisplayed to a user as an overlay image on the display.

Example 24 is an endoscopic system as in any of Examples 1-23, whereinthe overlay image is assigned a visible color for use on the display;wherein the visible color is 8-bit or 16-bit or n-bit.

Example 25 is an endoscopic system as in any of Examples 1-24, whereinthe light engine comprises a polarization filter.

Example 26 is an endoscopic system as in any of Examples 1-25, whereinthe polarization filter is located in a path of the electromagneticradiation.

Example 27 is an endoscopic system as in any of Examples 1-26, whereinthe polarization filter is located at a proximal end of the lumen.

Example 28 is an endoscopic system as in any of Examples 1-27, whereinthe polarization filter is located at a distal end of the lumen.

Example 29 is an endoscopic system as in any of Examples 1-28, whereinthe lens assembly comprises an electromagnetic radiation filter.

Example 30 is an endoscopic system as in any of Examples 1-29, whereinthe lens assembly comprises a polarization filter.

Example 31 is an endoscopic system as in any of Examples 1-30, whereineach pulse of electromagnetic radiation results in an exposure framecreated by the image sensor; wherein one or more exposure frames is fedto a corresponding system that will provide location of critical tissuestructures.

Example 32 is an endoscopic system as in any of Examples 1-31, whereinthe location of critical structures is received by the endoscopic systemand overlaid on a display, wherein the critical structures are encodedto any color selected by either an algorithm or a user.

It will be appreciated that various features disclosed herein providesignificant advantages and advancements in the art. The following claimsare exemplary of some of those features.

In the foregoing Detailed Description of the Disclosure, variousfeatures of the disclosure are grouped together in a single embodimentfor the purpose of streamlining the disclosure. This method ofdisclosure is not to be interpreted as reflecting an intention that theclaimed disclosure requires more features than are expressly recited ineach claim. Rather, inventive aspects lie in less than all features of asingle foregoing disclosed embodiment.

It is to be understood that the above-described arrangements are onlyillustrative of the application of the principles of the disclosure.Numerous modifications and alternative arrangements may be devised bythose skilled in the art without departing from the spirit and scope ofthe disclosure and the appended claims are intended to cover suchmodifications and arrangements.

Thus, while the disclosure has been shown in the drawings and describedabove with particularity and detail, it will be apparent to those ofordinary skill in the art that numerous modifications, including, butnot limited to, variations in size, materials, shape, form, function andmanner of operation, assembly and use may be made without departing fromthe principles and concepts set forth herein.

Further, where appropriate, functions described herein can be performedin one or more of: hardware, software, firmware, digital components, oranalog components. For example, one or more application specificintegrated circuits (ASICs) or field programmable gate arrays (FPGAs)can be programmed to carry out one or more of the systems and proceduresdescribed herein. Certain terms are used throughout the followingdescription and claims to refer to particular system components. As oneskilled in the art will appreciate, components may be referred to bydifferent names This document does not intend to distinguish betweencomponents that differ in name, but not function.

The foregoing description has been presented for the purposes ofillustration and description. It is not intended to be exhaustive or tolimit the disclosure to the precise form disclosed. Many modificationsand variations are possible in light of the above teaching. Further, itshould be noted that any or all the aforementioned alternateimplementations may be used in any combination desired to formadditional hybrid implementations of the disclosure.

Further, although specific implementations of the disclosure have beendescribed and illustrated, the disclosure is not to be limited to thespecific forms or arrangements of parts so described and illustrated.The scope of the disclosure is to be defined by the claims appendedhereto, any future claims submitted here and in different applications,and their equivalents.

1. An endoscopic system for use in a light deficient environmentcomprising: an imaging device comprising: a tube; one or more imagesensors; and a lens assembly comprising at least one optical elementcorresponding to said image sensor; a display for a user to visualize ascene; an image signal processing controller; and a light engine,wherein the light engine comprises: an illumination source generatingone or more pulses of electromagnetic radiation; and a lumentransmitting one or more pulses of electromagnetic radiation to a distaltip of an endoscope, wherein at least a portion of the one or morepulses of electromagnetic radiation includes an excitation wavelength ofelectromagnetic radiation between 770 nm and 790 nm that causes areagent to fluoresce at a wavelength that is different from theexcitation wavelength of the portion of the one or more pulses ofelectromagnetic radiation.
 2. The endoscopic system of claim 1, whereinthe system further comprises a filter that blocks the excitationwavelength of electromagnetic radiation between 770 nm and 790 nm. 3.The endoscopic system of claim 2, wherein the filter is located on theat least one optical element of the lens assembly, such that the filterblocks the excitation wavelength and allows a wavelength of thefluorescing reagent through the filter.
 4. The endoscopic system ofclaim 2, wherein each pulse of electromagnetic radiation results in anexposure frame created by the image sensor; wherein one or more exposureframes are displayed to a user as a single image on the display.
 5. Theendoscopic system of claim 4, wherein the single image is assigned avisible color for use on the display; wherein the visible color is 8-bitor 16-bit or n-bit.
 6. The endoscopic system of claim 2, wherein eachpulse of electromagnetic radiation results in an exposure frame createdby the image sensor; wherein one or more exposure frames are displayedto a user as an overlay image on the display.
 7. The endoscopic systemof claim 6, wherein the overlay image is assigned a visible color foruse on the display; wherein the visible color is 8-bit or 16-bit orn-bit.
 8. The endoscopic system of claim 1, wherein the image sensordetects a wavelength of the fluorescing reagent to provide an image ofone or more critical structures in a human body.
 9. The endoscopicsystem of claim 8, wherein the critical structures in a human bodyinclude one of a nerve, a ureter, a blood vessel, an artery, a bloodflow, and a tumor.
 10. The endoscopic system of claim 8, wherein the oneor more critical structures are cancer cells, and wherein the systemreceives fluoresced electromagnetic radiation from a molecule thatattaches a fluorophore that fluoresces when exposed to electromagneticradiation having a wavelength between 770 nm and 790 nm to one or moreof the cancer cells.
 11. The endoscopic system of claim 1, wherein eachpulse of electromagnetic radiation results in an exposure frame createdby the image sensor; wherein one or more exposure frames are displayedto a user as a single image on the display.
 12. The endoscopic system ofclaim 11, wherein the single image is assigned a visible color for useon the display; wherein the visible color is 8-bit or 16-bit or n-bit.13. The endoscopic system of claim 1, wherein each pulse ofelectromagnetic radiation results in an exposure frame created by theimage sensor; wherein one or more exposure frames are displayed to auser as an overlay image on the display.
 14. The endoscopic system ofclaim 13, wherein the overlay image is assigned a visible color for useon the display; wherein the visible color is 8-bit or 16-bit or n-bit.15. The endoscopic system of claim 1, wherein the illumination sourcegenerates one or more pulses of electromagnetic radiation at awavelength of 370 nm to 420 nm.
 16. The endoscopic system of claim 15,wherein each pulse of electromagnetic radiation results in an exposureframe created by the image sensor; wherein one or more exposure framesare displayed to a user as a single image on the display.
 17. Theendoscopic system of claim 16, wherein the single image is assigned avisible color for use on the display; wherein the visible color is 8-bitor 16-bit or n-bit.
 18. The endoscopic system of claim 15, wherein eachpulse of electromagnetic radiation results in an exposure frame createdby the image sensor; wherein one or more exposure frames are displayedto a user as an overlay image on the display.
 19. The endoscopic systemof claim 18, wherein the overlay image is assigned a visible color foruse on the display; wherein the visible color is 8-bit or 16-bit orn-bit.
 20. The endoscopic system of claim 1, wherein the illuminationsource generates one or more pulses of electromagnetic radiation at awavelength of 600 nm to 670 nm.
 21. The endoscopic system of claim 20,wherein each pulse of electromagnetic radiation results in an exposureframe created by the image sensor; wherein one or more exposure framesare displayed to a user as a single image on the display.
 22. Theendoscopic system of claim 21, wherein the single image is assigned avisible color for use on the display; wherein the visible color is 8-bitor 16-bit or n-bit.
 23. The endoscopic system of claim 20, wherein eachpulse of electromagnetic radiation results in an exposure frame createdby the image sensor; wherein one or more exposure frames are displayedto a user as an overlay image on the display.
 24. The endoscopic systemof claim 23, wherein the overlay image is assigned a visible color foruse on the display; wherein the visible color is 8-bit or 16-bit orn-bit.
 25. The endoscopic system of claim 1, wherein the light enginecomprises a polarization filter.
 26. The endoscopic system of claim 25,wherein the polarization filter is located in a path of theelectromagnetic radiation.
 27. The endoscopic system of claim 26,wherein the polarization filter is located at a proximal end of thelumen.
 28. The endoscopic system of claim 26, wherein the polarizationfilter is located at a distal end of the lumen.
 29. The endoscopicsystem of claim 1, wherein the lens assembly comprises anelectromagnetic radiation filter.
 30. The endoscopic system of claim 1,wherein the lens assembly comprises a polarization filter.
 31. Theendoscopic system of claim 1, wherein each pulse of electromagneticradiation results in an exposure frame created by the image sensor;wherein one or more exposure frames is fed to a corresponding systemthat will provide location of critical tissue structures.
 32. Theendoscopic system of claim 31, wherein the location of criticalstructures is received by the endoscopic system and overlaid on adisplay, wherein the critical structures are encoded to any colorselected by either an algorithm or a user.